Merge Scala and Java documentation

jlprat commented 8 years ago

This is not properly an issue, but more a discussion ticket.

I recently ported some documentation pages from the Scala docs to the Java docs. The content is basically 95% the same, and only the code snippets and the inline code examples differ. Would it make sense to think about some kind of abstraction so the main documentation text is shared?

Currently, if somebody changes something on a Scala documentation page, they need to also apply the same changes to the Java counterpart (if somebody forgets, both documentation will not be in sync).

2m commented 8 years ago

jlprat commented 8 years ago

Related to the docs, and how it is structured, right now it's complicated to find a specific topic as you need to navigate through the right sections till the right one (for example to get the page doc for a Directive). Is there any plan to add a search functionality?

ktoso commented 8 years ago

We have search on akka docs: scala_documentation_ _akka_documentation

and we'll enable the same for akka-http docs, once we have decided where it'll live. I'll have to talk to Algolia to set it up then.

jlprat commented 8 years ago

Cool! Yeah, I'm familiar with the Akka search, I didn't know it it will also appear for akka-http docs :)

ktoso commented 8 years ago

Absolutely, we'll also want it in alpakka et cetera. Hope Algolia will bear with us through this setting up :)

jrudolph commented 8 years ago

I've also suggested to merge Java and Scala documentation when possible. Maybe to get this started it would make sense to find out on which pages the biggest differences are because these would be the most difficult to merge.

ktoso commented 8 years ago

It used to be the case that directives were, but that's not that true anymore since we made the new DSL.
I think the most different things are marshalling right now. But those are a bit separate anyway, since they directly talk about "this is how to use jackson" etc, we could keep them separated.
We could get away with having one page nowadays perhaps since we have a simple switcher between scala/java code in paradox.

I think there should also be some "if I'm looking at a scala snippet, the description should use @@scala-snippet-description so we could have text change along with the snippet. This would allow us to really have 1 file, and only if needed explain more or less for a given lang. I'll open a ticket in paradox for this.

jrudolph commented 8 years ago

But isn't the tabbing independent from snippets anyway? So, you would manually create tabs for Java and Scala and just put in there whatever you need, code and description. I think the most important thing would then be to put the choice of language into a cookie, so that it will be remembered between different pages. I guess that can be implemented in Javascript.

jrudolph commented 8 years ago

And a script that switches all tabs on a page from Scala to Java or vice versa.

ktoso commented 8 years ago

I think the most important thing would then be to put the choice of language into a cookie, so that it will be remembered between different pages. I guess that can be implemented in Javascript.

Absolutely agree, let's see if they have a ticket for it already, I know it was discussed

ktoso commented 8 years ago

But isn't the tabbing independent from snippets anyway? So, you would manually create tabs for Java and Scala and just put in there whatever you need, code and description.

I guess that's true, but it would look a bit weird wouldn't it? Destroys the flow of reading a bit hm

jlprat commented 8 years ago

Don't forget the inline code snippets. Mainly, the only difference between the Scala and Java version is the generics syntax ([] vs <>)

ktoso commented 8 years ago

We should try to avoid those, as people then are confused which imports and context is needed for a snippet to work.

jlprat commented 8 years ago

I'm referring to this type of inline snippets:

The first route queries an asynchronous database and marshalls the Future[Option[Item]] result into a JSON response

From http://doc.akka.io/docs/akka-http/current/scala/http/introduction.html#routing-dsl-for-http-servers

ktoso commented 8 years ago

Ah, I see. True

jonas commented 8 years ago

Should try to also unifying the directive docs? With #11 we will likely end up duplicating docs in scaladoc and javadoc, but it would simplify a lot of things if we only extract from scaladoc. The main complication is interlinking between directives and linking to non-directive pages which requires parsing either javadoc or scaladoc markup.

jlprat commented 7 years ago

Is there any plan to have the docs converted to PDF? I find it quite useful to get a PDF version to read when offline BTW, there is no search on the docs, @ktoso did you talk to Algolia?

ktoso commented 7 years ago

BTW, there is no search on the docs, @ktoso did you talk to Algolia?

I just wrote them an email :)

jonas commented 7 years ago

@jlprat There's an upstream ticket to add PDF support to Paradox: lightbend/paradox#14 however this might need to wait until lightbend/paradox#81 is resolved.

I'll try to get back to hakking on Paradox to move things forward.

jrudolph commented 7 years ago

(Renamed the ticket to "Merge Scala and Java documentation")

I thought about a plan how to migrate the current documentation to a state where Java and Scala documentation is merged that would work in an incremental fashion without introducing too many conflicts in the transition period.

Here it is:

During the transition period, we create another node next to Java and Scala documentation just called "Documentation". The idea is to merge pages from the Java / Scala documentation to this new node. The new node should have the same document structure as the existing documentation.
In the first step, all pages in the new merged tree contain links to both the Scala and Java documentation.
Then over time pages are merged, one by one, replacing links to Scala/Java side with the actual merged contents. On the corresponding page on the Scala and Java side, a link to the merged page is placed. We never adapt any inner links, but those should always point to the merged page even if that is currently only a stub.
When all pages are merged, we remove the separate tree for Java and Scala.

We can decide not to merge some chapters (like directives) but we will need to think about how to represent these in the final tree.

During the process, the documentation will be a bit more confusing. We might think about introducing automatic redirection from stub pages to one of the links by making use of the cookie value that specifies the language preference of the user (which will be one of the things we need for the merged things anyway).

WDYT?

jlprat commented 7 years ago

If I understood your suggestion correctly, I think it might work. I only see a problem, the inline code snippets (or type signatures). It will not work easily when the inline contains a type parameter because of different syntax between Java and Scala, and sometimes the types are completely different. For example:

... which simply turns an HttpRequest instance into CompletionStage<HttpResponse>....

vs.

... which simply turns an HttpRequest instance into Future[HttpResponse]...

This is the only weak point I see.

jrudolph commented 7 years ago

My comment was more about the structural things so far.

For the actual content, I fully agree that we need to solve these situations. For this particular one, I'd suggest we introduce a new paradox directive which allows to specify inline parts in multiple languages. This directive should create multiple <span> with a language class set, so that javascript support can choose the right one based on the language selection.

jlprat commented 7 years ago

It sounds doable, but I agree with you that the migration process could be painful, or at least confusing.

+1 for the paradox feature :) I think @jonas said something about providing code snippets in multiple languages (like tab based code listings)

jrudolph commented 7 years ago

It seems that https://github.com/lightbend/paradox/pull/95 is starting to add features like that.

ktoso commented 7 years ago

Quick note that I agree Johannes's plan sounds very viable :)

jonas commented 7 years ago

Paradox's new overlay feature might come in handy, allowing the new "shared doc node" to be the fallback for the current Java/Scala.

http://developer.lightbend.com/docs/paradox/latest/features/overlay.html

jonas commented 7 years ago

Here's a proof of concept of using overlays for Akka HTTP docs: https://github.com/jonas/akka-http/commit/f36a026720e1ae84f477bddd1035cd740ca76067

Source directory:

> ls -l docs/src/main/paradox/
total 0
drwxr-xr-x  4 fonseca  wheel  136 20 Feb 11:09 _template
drwxr-xr-x  3 fonseca  wheel  102  5 Mar 20:04 common
drwxr-xr-x  3 fonseca  wheel  102 15 Oct 09:29 java
drwxr-xr-x  3 fonseca  wheel  102 15 Oct 09:29 scala

Target structure:

 > ls -l docs/target/paradox/site
total 0
drwxr-xr-x  6 fonseca  wheel  204  5 Mar 17:59 java
drwxr-xr-x  6 fonseca  wheel  204  5 Mar 17:42 scala

It "firewalls" Java and Scala docs so it is not possible to interlink using @ref anymore which ensures we no longer link to Scala pages from Java. Anyway, overlays would allow us to generate java and scala versions of the documentation very easily while migrating stuff to a common folder and provides the choice of keeping docs separate.

jrudolph commented 7 years ago

Thanks looks interesting, will have a look once I have a bit more time.

raboof commented 7 years ago

@jonas' proposal looks like a good use of the overlays to arrive at a gradual way of arriving at a single common documentation base. Looks like this would be a slight variation on @jrudolph's proposal above:

Introduce a (nearly-empty) 'common' tree next to 'scala' and 'java' that the 'scala' and 'java' trees fall back to (this is basically what @jonas's branch achieves)
Add UI to switch between java/scala versions of the docs

At this point this can be published/merged. Then:

Make the tree more consistent between Java and Scala versions, perhaps move some pages to 'common' already
Add a directive to paradox to switch between variations (for us: java/scala) within one document
Gradually move more things to 'common'

This will take a while. Then when everything has been moved to 'common' we can add support to make the language switching more 'dynamic' (without a page redirect) and publish 1 tree instead of 2 again, perhaps even removing the need to use overlays entirely.

raboof commented 7 years ago

On the other hand going straight for the 'merged' end-state might save some 'throwaway' work.

I looked into client-side 'group switching' in https://github.com/lightbend/paradox/pull/107, seems doable.

jrudolph commented 7 years ago

Let's focus on just converting a single page with all the needed paradox features to get the ball rolling. I also think that all the steps to get there are doable, but we might need to push a bit do get them done soon.

The problem I see with merging all pages at once, is the sheer volume. Even without the directive pages, there is lots of content and just reviewing a huge bulk of documents seems like quite a burden. Maybe it's not clear with what you mean with "going straight" for the end-state. It sounds a bit like the move should be done in a single atomic operation which might mean that only person is involved in doing the change? That's something I'd like to avoid (and maybe it's not what you meant). At the current point I would also be fine with a feature branch for the conversion to be merged to master once the conversion is complete. The problem with that is that it invites conflicts if much work is done on the documentation content while we do the conversion. When we started the conversation that was the case so I wanted to prevent this disadvantage of a long-running feature branch. Right now the amount of incoming documentation changes isn't too high so it might make sense to do it in a branch. WDYT?

raboof commented 7 years ago

With "going straight for the end-state" I meant skipping the 'overlays' approach and merging the docs into one tree immediately. Indeed using a feature branch we can split up the work, but merging it would be an atomic operation which is still a disadvantage.

I'm a bit on the fence on this one still, perhaps we should indeed just try converting one page 'completely' and then see how that looks.

raboof commented 7 years ago

First 'merged' page test: http://pi.bzzt.net/akka-docs-new/guide/quickstart.html (based on https://github.com/akka/akka/compare/integrate-docs-scala-java)

jlprat commented 7 years ago

That looks quite cool :)

jrudolph commented 7 years ago

I agree, very nice :) I think we should try to improve the inline snippet syntax. Can't we just introduce another special cased tag that allows to write that in a more concise way?

ktoso commented 7 years ago

Looks awesome @raboof :-) I think Peter proposed some nice syntax in the paradox repo, like @scalaOnly... or @scalaJava etc

He-Pin commented 7 years ago

@raboof Wow, it's really great! I think I could make use of paradox later too.

jlprat commented 7 years ago

Shall we close this one in favor of #1290 ?

raboof commented 7 years ago

I agree that makes sense

akka / akka-http

Merge Scala and Java documentation #527