Write release notes for v5.0.0

RubenVerborgh commented 5 years ago

Include upgrade notes (see #946)

rubensworks commented 5 years ago

I assume this is for the CHANGELOG.md? How is this file currently updated? Manually, or with some tool?

kjetilk commented 5 years ago

Yeah, I think we can put it in there. Right now, it isn't updated. I spent an hour trying to hack a tool so that we could use the DOAP changeset vocab to maintain the changelog, like I do with my Perl modules, but it wasn't that straightforward to extract out of the toolset.

rubensworks commented 5 years ago

the DOAP changeset vocab to maintain the changelog, like I do with my Perl modules

Oh, nice :-)

but it wasn't that straightforward to extract out of the toolset

I recently created manual-git-changelog for automating changelogs as much as possible in my projects. Not sure exactly what the release workflow for NSS is exactly, but if it matches with my workflow, then this might be a useful tool to use here.

melvincarvalho commented 5 years ago

Pod owners also require a script, or document, explaining back out the file system changes.

See :

https://github.com/solid/node-solid-server/issues/950#issuecomment-440562194

RubenVerborgh commented 5 years ago

That's not planned at the moment; we first need evidence that more than one person in the community needs this. This change has been pending for over a year and no one else reacted or proposed a timely alternative, so I'm confident we're good.

melvincarvalho commented 5 years ago

@RubenVerborgh understood, but disappointed. For the record, I strongly think this should have been engineered as a switch or config option. Also the file system changes should have been in its own (major) version.

As a pod owner, the upgrade performs changes to every user's file system, without their control or consent, and without a clear way to back that out. It also breaks compatibility with every other solid server out there including every version of node solid server other than 5.0.0. This is not a change to be taken lightly.

With my solid.community hat on, we will wait until this change is considered stable on other pods, and back out instructions are established, before moving to this version. My estimate is that 1-2 weeks after the first large pod moves, we should be ready. Any and all documentation will be extremely helpful.

rubensworks commented 5 years ago

Just my two cents:

For the record, I strongly think this should have been engineered as a switch or config option.

This would mean that two systems need to be maintained, which requires additional effort and becomes an additional source of bugs.

Also the file system changes should have been in its own version.

5.0.0 is a major update, so IMO this is the perfect time for this transition :-)

It also breaks compatibility with every other solid server out there including every version of node solid server other than 5.0.0.

Hence the major version change.

melvincarvalho commented 5 years ago

@rubensworks thanks for the comments, you make good points imho. Just on one, which I think is relevant to this thread.

This would mean that two systems need to be maintained, which requires additional effort and becomes an additional source of bugs.

BTW This is pretty normal in the node ecosystem. For example, node 8 is supported for quite some time after the release of node 10. Similarly with ubuntu etc.

What is happening here is that a cliff edge is being introduced with little or no concern for those running pods. My point is that in such a scenario, documentation becomes all the more valuable.

RubenVerborgh commented 5 years ago

@melvincarvalho Here's how discussion works. Either you reply to my technical arguments and you bring evidence that >1 person is affected, or I refrain from further discussion and just move on as planned.

Since you have not replied to my technical points in the other thread, they still stand and there is no value in continuing.

melvincarvalho commented 5 years ago

@RubenVerborgh understood. Thank you for your time and responses. In general, my posts were not specifically directed at you.

melvincarvalho commented 5 years ago

I think it's worth noting a few points. At least here, if not in the notes.

This version of node solid server loses some desirable properties.

One great things about the web and solid is that you just point a web server at a file system and you are up and running. You point an LDP server at a file system and you get more features. You point a solid server at a file system and you get more features still.

I wrote the following on introduction to solid

Solid is a set of modular specifications, which build on, and extend the founding technology of the world wide web (HTTP, REST, HTML). They are 100% backwards compatible with the existing web. Each spec, taken in isolation, provides extra features to an existing system. However, when used in combination, they enable exciting new possibilities for web sites and applications.

IMHO 5.0.0 doesnt fully uphold that.

In designing read write technology Im often guided by the following princple:

The way the Web spread was a piece at a time. So you could take html without taking http. So the failure of NEXT was a lesson, don’t try to sell it all at one time. Sell each piece on its own merits. Never insist that everybody take all. They will take all the pieces once they see how it fits together. -- Tim Berners-Lee

Again Im unsure that this is fully upheld.

The main issues, as I see it, with using solid as a web server on top of a file system are :

It breaks compatibility with node solid servers below v5
It breaks compatibility with other solid servers
It breaks compatibility with LDP servers
It breaks compatibility with web servers
Almost every profile is affected

Looking on the positive side, an important edge case is fixed (preserving content type for extensionless files) and this is something that has been on the wish list for a long time. Additionally, it's coded in such a way that allows flexibility and ability to back out the changes without too much effort (well done on this point!).

As a stop gap this may well be the sweet spot as a way to get a server working with all the properties that we want. And in a time frame that allows the eco system to grow.

What I think we can look at post v5, is that we will have a relatively bug free implementation and have the ability to improve the way some things are done, for v6.0 and above. One thing we could look at is other mechanisms to fix this, such as to put the content type in a .meta file, or some other innovation of this kind.

melvincarvalho commented 5 years ago

I think the release notes should also say something about changes to the way certain files are created, such as profile/card and append only POSTS (such as to an inbox).

csarven commented 5 years ago

This version of node solid server loses some desirable properties.

It is fixing what was previously malformed. We can debate about a well-document bug being a feature I suppose.. but that'd be a waste of time.

It breaks compatibility with node solid servers below v5

That's literally what constitutes a major version.

It breaks compatibility with other solid servers

There is no specified or implemented server-server communication at the moment. Citation needed.

It breaks compatibility with LDP servers

Citation needed.

It breaks compatibility with web servers

Citation needed. Moreover, there was no promise at any point in time that "bridging" (in the general sense of the word) was on the agenda of delivering a "stable" version of NSS.

Almost every profile is affected

Citation needed. For who exactly? The contrary is demonstrated as I see it.

Above all, is there any hard evidence here? Who is ultimately affected? Is there a survey with the admins of running NSS? Can you share the data?

What I think we can look at post v5, is that we will have a relatively bug free implementation and have the ability to improve the way some things are done, for v6.0 and above.

Define "bug free". The point of v5 is that it is "bug free" re URI/file mapping. Anything that relied on previous behaviour was deemed to be a bug to begin with, not to forget that any supplemental processes relying on that bug is completely out of scope of NSS.

One thing we could look at is other mechanisms to fix this, such as to put the content type in a .meta file, or some other innovation of this kind.

.meta and alternative approaches have been discussed. Sure, those alternatives may in fact be ultimately better than what's in front of us right now, however that is completely hypothetical. Those that wish to pursue the alternatives further discuss and present PRs, as opposed to dismissing what went through a community process. Until than, we have a clear and tangible step forward. Today.

There is absolutely nothing new here as there was plenty of time to discuss and develop. It is progress by every reasonable measure. If alternative approaches were in any way better or bridging or smooth transitions to other systems - whatever you want to call it - was important, then those that cared enough should make an effort to develop it - or better yet, should have done it by now. Certainly not on the eve of making a release.

Attempting to stall a release, in "Write release notes for v5.0.0" of all places, is totally uncool IMHO.

rubensworks commented 5 years ago

Migration steps have been added in #967.

Shall we leave this issue open for any other things that need to be added to the changelog for release 5.0.0?

kjetilk commented 5 years ago

Shall we leave this issue open for any other things that need to be added to the changelog for release 5.0.0?

Yup.

melvincarvalho commented 5 years ago

@csarven

Attempting to stall a release, in "Write release notes for v5.0.0" of all places, is totally uncool IMHO.

I went out of my way to ensure that my notes were added in note form

I think it's worth noting a few points. At least here, if not in the notes.

And also stated that this would not stall the release

As a stop gap this may well be the sweet spot as a way to get a server working with all the properties that we want. And in a time frame that allows the eco system to grow.

Disagreements are fine, but ad hominems, attributing motives to individuals, or passive aggressive jabs are not. I know you can do better. I would invite you to review our code of conduct. We would like to make github a welcoming and constructive place to discuss all things solid.

RubenVerborgh commented 5 years ago

@melvincarvalho My problem with this discussion is as follows.

First of all, I assume we all mean well. So what I will be sketching below is not the result of bad intent, but is a result nonetheless.

Unfortunately, many of the discussions that we have follow this pattern:

person A: opinion or concern not substantiated by technical arguments
person B: technical arguments that aim to indicate the concern is not an issue
person A: more opinions or concerns not substantiated by technical arguments
person B: technical arguments that aim to indicate the concerns are not an issue
…
for lack of technical arguments, the concern of person A is not addressed. The concern might or might not be valid, but it is impossible to tell for lack of evidence and arguments, so it cannot be taken into account.

The problematic thing in the discussion above is that the technical arguments of person B are ignored, not replied to, and as such, there is no evidence to take the opinion of person A into account.

Here is a more ideal discussion structure:

person A voices an opinion or concern, backed up by technical arguments
person B brings more technical arguments to the table that shed a different light on the concern, explains why some of the technical arguments of A are not valid
person A explains that person B has missed a couple of cases, supported by technical arguments
…
Based on technical arguments, a decision is made.

You can see these discussion patterns happening in the current thread, #950, https://github.com/solid/vocab/issues/31, #788, and many more.

All we are asking, is to back up your point with technical arguments, and to reply to our technical arguments with technical arguments. Anything else is, unfortunately, a waste of time, because decisions in this project are taken based on technical arguments.

I'm not saying that your points are invalid, I'm just saying that they are not backed up by technical arguments. So please, add technical arguments and/or ask others to help you do so, such that this discussion has a potentially different outcome. Discussing without arguments will not alter the outcome of a discussion, and is as such pointless.

For me, this comes back to the first point of our Code of Conduct: respect, professionalism, fairness. Have respect for our technical arguments, that we are trying to make in order for you to phrase your point better. Be professional: make technical arguments that can actually influence the discussion. Fair: if people invest time in writing replies, reply to their points.

On-topic post below.

RubenVerborgh commented 5 years ago

@melvincarvalho In the post below, I will invite you to make specific technical arguments. As per my post above, please accept these invitations; only technical arguments can alter the outcome of this discussion.

This version of node solid server loses some desirable properties.

I disagree for reasons state above and in #950, which have not been replied to. Please explain which properties and why.

One great things about the web and solid is that you just point a web server at a file system and you are up and running.

I disagree for reasons state above and in #950 (awaiting reply). Please explain with technical arguments why this would not be the case. Please give me one example of a valid filesystem that will not be up and running with v5, but that was up and running with v4.

IMHO 5.0.0 doesnt fully uphold that.

Please explain with technical arguments why this would not be the case. I disagree (above and #950); no replies yet.

Again Im unsure that this is fully upheld.

Please explain with technical arguments why this would not be the case. I disagree (above and #950); no replies yet.

* It breaks compatibility with node solid servers below v5

Please explain with technical arguments why this would not be the case.

* It breaks compatibility with other solid servers

This is not correct. Please explain.

* It breaks compatibility with LDP servers

This is not correct.

* It breaks compatibility with web servers

This is not correct.

* Almost every profile is affected

This is not correct.

So @melvincarvalho, please, I'm really begging at this point: can we please break the cycle? Please add arguments for the statements above, most of which can easily be proven wrong. Stating them again doesn't make them valid; only technical arguments might.

In absence of such arguments, we unfortunately are unable take these statements into account.

melvincarvalho commented 5 years ago

@RubenVerborgh I actually thought we had resolved this. We discussed 4 possible alternatives, and eventually the 4th (knowledge of how to back out the change) was exchanged, and I was content with that. It's not always possible to field every point on github without bloating an issue. We discussed further points on email, and I would be happy to address outstanding concerns.

I dont have an issue with technical disagreements. What I dont accept are ad hominems, attributing motives to individuals, or passive aggressive jabs as per @csarven comment as follows :

Attempting to stall a release, in "Write release notes for v5.0.0" of all places, is totally uncool IMHO.

I raised this with the individual initially, then with the inrupt community manager, who said they were too busy to deal with it. So I decided to resolve it myself, as it is clear cut.

Be professional: make technical arguments that can actually influence the discussion. .

Just fielding this. Thank you for pointing me to the code of conduct. I always try to be respectful and professional. That you would think otherwise, and choose to state that publicly I find disappointing. I am sure that you know, that were the roles reversed I would not do the same.

RubenVerborgh commented 5 years ago

I always try to be respectful and professional.

No doubt on my mind, see the very first point in my post, which I made for this purpose.

My message is not that you're not trying, but that, in my opinion, you're currently not succeeding.

That you would think otherwise, and choose to state that publicly I find disappointing.

The very fact that you, again, did not reply to any of my technical points despite my explicit invitation, indicates the necessity of that action.

melvincarvalho commented 5 years ago

The very fact that you, again, did not reply to any of my technical points despite my explicit invitation, indicates the necessity of that action.

Somewhat vague. In your response you answered your own questions. If there is one of more you feel have not responded to, it is helpful to either provide a reference, or perhaps kindly restate it.

However you are completely missing the point.

I am not stating a technical objection. I am riling against a personal attack. Which you decided to exacerbate. Your silence is deafening.

RubenVerborgh commented 5 years ago

I am not stating a technical objection.

And that's exactly our problem. We really wish you did.

I believe @csarven and I used different words to explain the same thing: your concerns lack arguments, so we cannot act upon them. Hence, discussing only costs time but does not effect change, and thus—unintentionally—stalls a release. As such, I do not consider this a personal attack, but a correct characterization of the fruitless growth of the discussion.

I don't understand why you are unwilling or unable to provide technical arguments, despite explicit requests. A search for "please" within this page will show just a subset of the many you haven't replied to and—if the past may serve as a predictor for the present—never will. This is why people's patience is running out. Argue properly, or let us make progress.

RubenVerborgh commented 5 years ago

Note: @melvincarvalho has been asked to create new issues for every concern that needs to be addressed. A first one is https://github.com/solid/node-solid-server/issues/975, where we discuss whether v5 breaks LDP compatibility.

In order to bring clarity to this thread, and since we have a better mechanism for following up on individual concerns, I will hide all comments that are not directly relevant to "Write release notes", including this comment. Note that these comments are not being deleted.

rubensworks commented 5 years ago

For reference, we should also mention the removal of the -v option in the changelog.

rubensworks commented 5 years ago

Another one: config options (both CLI and config.json) are now kebab-cased instead of camelCased.

kjetilk commented 5 years ago

Just to record the PRs that may be worth mentioning for later review:

1014
the whole trusted origin stuff, #1012 #770
ToC, #989,
ACL: #984, #970, #969, #786
logout fixes, #957 #927
ldp:inbox, #794
acl:default #768
767

kjetilk commented 5 years ago

I think the release notes should also say something about changes to the way certain files are created, such as profile/card and append only POSTS (such as to an inbox).

Not sure what you mean, @melvincarvalho . Could you suggest a short sentence that is suitable?

kjetilk commented 5 years ago

I think we're pretty much good to go with these as release notes. We should probably write more friendly documentation with 5.0.0 as starting point, but that's a different matter.

I'm going to keep the issue open until we actually release 5.0.0 though, in case there are any new things that hop up.

kjetilk commented 5 years ago

Come to think of: We need a note on updating files in the POD in the upgrade instructions, right?

megoth commented 5 years ago

I think we also need to mention the commands invalidusernames and updateindex somewhere; maybe not the release notes, but I don't think we've documented this beside the help texts in the CLI.

We should also note that before doing updateindex the first time, the POD owners should be notified that they need to change their index.html if they want to keep the old one (a tutorial for this is available at https://github.com/megoth/solid-update-index-tutorial).

RubenVerborgh commented 5 years ago

I thought nothing changed for pod owners? Old HTML doesn’t have the meta tag, so should’ve stay the same.

melvincarvalho commented 5 years ago

Not sure what you mean, @melvincarvalho

@kjetilk are you aware that files created on the server such as card will no longer be named card, but rather, card$.ttl (iirc), in version 5?

I think that's not the only case and applies in other areas such as inboxes.

This is a change from other versions of node solid server, so it might be worth letting people know?

megoth commented 5 years ago

I thought nothing changed for pod owners? Old HTML doesn’t have the meta tag, so should’ve stay the same.

That is not the way the script has been written, there are three scenarios:

No meta-tag for solid-allow-automatic-updates --> do update
solid-allow-automatic-updates is set to true --> do update
solid-allow-automatic-updates is set to false (or anything beside true really) --> do not update

Point 1 should maybe be "do not update"? It's very easy to fix, if that's the case, but I assumed most people, also people with older PODs wanted to update their index.html unless stated otherwise? I'm sorry if this was faulty assumption.

kjetilk commented 5 years ago

Not sure what you mean, @melvincarvalho

@kjetilk are you aware that files created on the server such as card will no longer be named card, but rather, card$.ttl (iirc), in version 5?

I think that's not the only case and applies in other areas such as inboxes.

This is a change from other versions of node solid server, so it might be worth letting people know?

Oh yes, but that's in there with an extensive description. I thought you meant that there were changes to the HTTP interface, but lost the context there. So, that should be OK.

RubenVerborgh commented 5 years ago

1. No meta-tag for `solid-allow-automatic-updates` --> do update

My opinion is that we shouldn't touch it—perhaps unless it is identical to the previous template (but the check for that is a bit more involved, as it involves filling out fields in that template).

melvincarvalho commented 5 years ago

I'm good with the script changing the system generated index.html file, but only the index.html file. A few reasons for this.

Eventually I think the index file will go away and be replaced with a client side app, I think
It was only intended as a temporary stop gap I believe
People have already complained that it was showing email addresses, and I think that is a valid concern
There is an opt out mechanism

It strikes me that pod admins, will ultimately make that decision on a per server basis, in the spirit of owning your own data. It may be possible to add a swtich for this, or that might be overkill

melvincarvalho commented 5 years ago

perhaps unless it is identical to the previous template

This is a nice idea!

melvincarvalho commented 5 years ago

the check for that is a bit more involved, as it involves filling out fields in that template

That would do it. However perhaps another heuristic could be to check the time stamp relative to another system generated file, to ensure it hasnt changed.

Or maybe even created time vs modified time? I dont know how reliable that is.

megoth commented 5 years ago

the check for that is a bit more involved, as it involves filling out fields in that template

That would do it. However perhaps another heuristic could be to check the time stamp relative to another system generated file, to ensure it hasnt changed.

Or maybe even created time vs modified time? I dont know how reliable that is.

I've created https://github.com/solid/node-solid-server/issues/1030 for discussing this issue further, as it kind of strays from the main topic of this issue.

kjetilk commented 5 years ago

I noticed some differences to the server setup that wasn't on my radar:

  "emailHost": 
  "emailPort": 
 "emailAuthUser": 
  "emailAuthPass":

I think we need to have them in the changelog.

michielbdejong commented 5 years ago

part of #1145

nodeSolidServer / node-solid-server

Write release notes for v5.0.0 #951

1014

767