search

purescript / pursuit

Website for hosting and searching PureScript API documentation
https://pursuit.purescript.org/
Other
169 stars 47 forks source link

Give up on package uploads #393

Open klntsky opened 5 years ago

klntsky commented 5 years ago

One way to resolve issues similar to #392, #382 and #335 once and for all is to give up on the ability to upload packages directly and just reuse package-sets as a source for docs generation: it is already a requirement for all packages in package-sets to be published on bower.

Moreover, doing so will make impossible the situation when some package is not found in the bower registry, but its docs are still on pursuit (obviously, this will be true up to delays between SearchIndex updates).

Currently there are three "registries": bower, pursuit and package-sets, and the latter two are only partially intersecting. This situation is worse than having only two, with one containing a subset of packages of the another.

The proposed change will make it harder to publish documentation on pursuit, as it will require making a PR to package-sets, but OTOH we'll be able to guarantee that if something is found on pursuit, it will just work, which would be beneficial for all, especially for new users of the language.

Additionally, the change will open a straightforward way to implement #267 .

hdgarrood commented 5 years ago

Sorry, but I would rather not do this. Package sets are great, but I feel quite strongly that they should not be the be-all and end-all of package management. It should still be possible for packages to get their docs up on Pursuit before those packages make it into the official package set, for example, and it should also be a valid option for package authors to opt out of using package sets entirely (e.g. if they do not expect to be able to keep their package up to date).

Moreover, doing so will make impossible the situation when some package is not found in the bower registry, but its docs are still on pursuit (obviously, this will be true up to delays between SearchIndex updates).

I would rather address this (and also #335) by doing something along the lines of checking on the server on package upload that the package exists in the bower registry and that its repository matches the one specified in the package manifest file.

Currently there are three "registries": bower, pursuit and package-sets, and the latter two are only partially intersecting. This situation is worse than having only two, with one containing a subset of packages of the another.

I agree. Bower is "the" registry; anything which exists in the other two should be the same as it is in Bower. When you say the latter two are only partially intersecting, do you mean that there are package names which refer to different repos in pursuit versus in package-sets? Or just that there are some which exist in one but not the other? I think the latter is inevitable to some extent.

Periodically checking that everything in the official package set is uploaded to Pursuit would be great though.

klntsky commented 5 years ago

it should also be a valid option for package authors to opt out of using package sets entirely (e.g. if they do not expect to be able to keep their package up to date).

I agree that providing reproducible builds and hosting documentation are two separate concerns, but I also see the dead docs/search results as a problem for the users (which can be solved by sticking to package sets).

if they do not expect to be able to keep their package up to date

In that case, do we really want to keep the docs available?

do you mean that there are package names which refer to different repos in pursuit versus in package-sets? Or just that there are some which exist in one but not the other?

I meant the latter, though the former is also possible.

I think the latter is inevitable to some extent.

Sorry, but I don't currently see why it is inevitable: if there were only one registry, no inconsistency could occur.

f-f commented 5 years ago

I don't think this is a great idea as it is currently formulated, but I think there's great value in exploring this space (typing down this comment was a great trip, I started with "I don't have an opinion on this" 😅)

Package sets are amazing, but as @hdgarrood said people should have the option to just opt-out of the official package set (or any package set), because keeping your package in there requires maintenance and this might not be fine for everyone. So as in Haskell we have Hackage, I think Pursuit is fulfilling well that role for PureScript. However a difference with Hackage is that while that's "the registry" for Haskell, in our case that's Bower and not Pursuit. The problem here seems to be that Bower and Pursuit get out of sync, so maybe that's the main thing we should focus on?

An idea for solving that would be to actually "give up on package uploads" and use Bower as a feed: as new packages prefixed with purescript- would come in (or new tags for the existing ones are uploaded), a big CI would generate docs for that package/tag and add them to the Pursuit DB. This would solve several issues at once:

The only downside I see here is that the CI will need to be coded and be run somewhere, but it shouldn't be a big deal to have someone donate some computing power (e.g. I have @spacchettibotti on a machine at work)

I'll note that I'm not inventing this idea, but this is exactly what cljdoc does: it watches Clojars (the main place for publishing Clojure libs) for new uploads, and builds docs for them. The community is very happy about it.

Another (but orthogonal IMHO) problem @klntsky mentioned is giving a good experience to newcomers and guaranteeing that packages will "just work". I have this issue myself all the time with Hackage, and Stackage makes it much more pleasant. So it could be a good idea to generate docs for every package set (note: in addition to Pursuit, not instead of) and point newcomers to them. And in fact the goal of https://github.com/spacchetti/spago/issues/89 (on which @hdgarrood has been of great great help) is exactly that: generate a "static Pursuit" (that is, html docs with no backend but with client-side search) for your project, which means also every package set (which is just a project with all packages in the set as dependencies). At this point there's only PureScript to be written there - i.e. the only thing missing is the search index and client-side search - so if someone wants to help that would be great 😄

hdgarrood commented 5 years ago

if they do not expect to be able to keep their package up to date

In that case, do we really want to keep the docs available?

Yes, definitely. Using the latest versions of everything as far as you can is a valid approach, but so is pinning a set of dependencies and just using those for a long time. Just because a package doesn't build with the latest versions of everything it doesn't mean that people aren't still deriving value from it.

do you mean that there are package names which refer to different repos in pursuit versus in package-sets? Or just that there are some which exist in one but not the other?

I meant the latter, though the former is also possible.

I think the latter is inevitable to some extent.

Sorry, but I don't currently see why it is inevitable: if there were only one registry, no inconsistency could occur.

It's inevitable that some packages will exist in Bower and Pursuit but not package-sets, because authors might not want to keep updating them. If a package were in package-sets but not in Pursuit, the only reason I can think of for that is that the author forgot to upload the docs to Pursuit, or something went wrong in the process of generating and uploading them. This is indeed something we can do a bit more work to address and I think it would be good to try to address it. In particular, having a kind of CI server/recurring job to help keep Pursuit docs up to date like what @f-f is describing does sound like it would indeed be a good thing to have.

Combining bower and pursuit into one single registry means being responsible for package uploads and package distribution, and I'm pretty confident that it would not be a good idea for us to take on those responsibilities (with the resources we currently have).