Formatable changes log (CHANGES.txt is easy to edit but not so friendly to read by Lucene users) [LUCENE-1157]

[asfimport]

Background in http://www.nabble.com/formatable-changes-log-tt15078749.html

---

Migrated from LUCENE-1157 by Doron Cohen, resolved Apr 20 2008 Attachments: lucene-1157.patch, lucene-1157-take2.patch, lucene-1157-take3.patch Linked issues:

• 2507

  • 2333

  • 2345

[asfimport]

Doron Cohen (migrated from JIRA)

Patch based on Steven A Row's perl script (see http://www.nabble.com/formatable-changes-log-tt15078749.html).

Changes:

• new target "changes-to-html" in common-build generates Changes.html if CHANGES.txt was modified.
• compile-core depends on this target.
• two stylesheets in Changes.html - fancy and simple - something to start with.
• only showing trunk and last official release, older releases are under "older".
• counters for each section tell you how many items (e.g. how many bug fixes).
• site links to Changes.html of svn head.

Note:

• The site link won't work until this is committed.
• Linking to svn-trunk-head-Changes.html makes committing modifications to CHANGES.txt no more tedious than before - only difference is: checkin alsoChanges.html.

See http://people.apache.org/\~doronc/changes2/Changes.html

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Attached lucene-1157-take2.patch, which includes all of Doron's patch, and merges in some changes I made to changes2html.pl:

• added links to JIRA versions of Bugzilla bugs;
• converted most lists to be numbered instead of bulleted, in preparation for the new non-numbered CHANGES.txt style;
• moved large variable definitions to subroutines at the bottom of the script;
• now escaping HTML metachars '<', '>' and '&' in the output; and
• reformatted to approximate Lucene's coding convention.

This patch also includes an updated Changes.html.

[asfimport]

Daniel Naber (migrated from JIRA)

It would be nice to have this working with Javascript disabled, i.e. to have all items expanded by default in that case. This could be done by displaying all items by default and adding this code at the bottom:

<SCRIPT> for (var i = 0; i <document.getElementsByTagName("ol").length; i++) { document.getElementsByTagName("ol")[i].style.display = "none"; } </SCRIPT>

Not very clean, but I don't know a better solution for now.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Hi Daniel,

I agree that the disabled/unavailable javascript scenario should be handled.

The attached lucene-1157-take3.patch puts your code, with some caveats, in a function which is then assigned to window.onload.

I tested it on Firefox 2.0, Opera 9.24, and Internet Explorer 6 and 7, all on Windows.

This patch includes all files in the previous takes.

[asfimport]

Doron Cohen (migrated from JIRA)

It looks great - and works both with javascript enabled and disabled (I checked only on Windows though). Links to both Bugzilla and JIRA work too. I will commit this soon unless there are objections.

converted most lists to be numbered instead of bulleted, in preparation for the new non-numbered CHANGES.txt style;

I anticipate that some would prefer to preserve numbering in CHANGES.txt (though but the pl script works either way).

I think it should not be too difficult now to create a second pl script, based on this one, that would generate the status.xml file for Forrest, in case we also want that type of presentation...

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

I think it should not be too difficult now to create a second pl script, based on this one, that would generate the status.xml file for Forrest, in case we also want that type of presentation...

Yes, I agree - the parsing of CHANGES.txt is separate from the output code, so only new output code would be required.

[asfimport]

Doron Cohen (migrated from JIRA)

Ok I checked in the creation of Changes.html from changes.txt. thanks Steven!

The Web site update part seems trickier than I thought.

• Adding a link in the site to http://svn.apache.org/viewvc/lucene/java/trunk/Changes.html?view=co does not work so well, because of the way that page is served by ViewVC.
• Linking to http://svn.apache.org/repos/asf/lucene/java/trunk/Changes.html isn't working either because svn returns the source of that file.
• In addition there are unidentifiable characters in Changes.html. They are also in CHANGES.txt. I'm sure I read something about why they are added but cannot find it now.

Ideas?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

According to http://wiki.apache.org/lucene-java/HowToUpdateTheWebsite , anything checked into trunk/docs/ will be automatically mirrored to the live website by a cron job running under Grant's account.

So, wouldn't it work to have Changes.html (and the stylesheets too) live in trunk/docs/ ?

[asfimport]

Doron Cohen (migrated from JIRA)

Seems that Changes.html sould not be in svn at all. Instead, it should have same status as javadocs - both are generated documentation. Instead of creating it as part of compile-core I'll create it as part of javadocs-core. Instead of created as part of committing, it would be created as part of nightly build, and copied to the site by Grant's scripts. I'll go on with this tomorrow.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

there are unidentifiable characters in Changes.html. They are also in CHANGES.txt. I'm sure I read something about why they are added but cannot find it now.

The first three bytes of CHANGES.txt are a UTF-8 BOM (byte-order mark). In Unicode's fixed-width encodings, e.g. UTF-16, the character U+FEFF is reserved for the beginnings of streams to denote the endian-ness of the character serialization.

UTF-8 is non-endian (invariant byte order given a character); the use of the BOM in UTF-8, where it is serialized as three bytes, is solely to indicate that the encoding of the stream is UTF-8.

Microsoft's tools like to put BOMs at the beginnings of UTF-8 encoded files.

[asfimport]

Doron Cohen (migrated from JIRA)

So, wouldn't it work to have Changes.html (and the stylesheets too) live in trunk/docs/ ?

Yes I agree, they should move so that Grant's job copies them. But I would like to make them part of the javadocs, so that there's no need recompile with each change and no need to check-in Changes.html. I'll revert this and continue tomorrow.

[asfimport]

Doron Cohen (migrated from JIRA)

Will this work? (Grant, others...)

• Changes.html not in svn, only dynamically created.
• The two css files are in svn, under docs/changes.
• The perl file is in svn, under src/site/changes.
• A new target "changes-to-html" writes to build/docs/changes/Changes.html
• The script that invokes "ant javadocs" and copies them to the nightly site is modified to also invoke "ant changes-to-html" and copy that too.

Looking at nightly.sh it is not clear how to proceed b/c all the scp's are commented out (probably because Hudson now does this?)

Thanks, Doron

[asfimport]

Doron Cohen (migrated from JIRA)

Microsoft's tools like to put BOMs at the beginnings of UTF-8 encoded files.

Thanks Steven, yes this is it. More info on this in http://en.wikipedia.org/wiki/Byte_Order_Mark We can filter this out when creating the html file. Seems this was recently added to the file in r615329 - I will remove it but it will return.. we can print a warning at the end of build if CHANGES.txt contains this BOMs.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

Here's how the main website gets published, from my crontab:

!/bin/sh

/usr/local/bin/svn export --force http://svn.apache.org/repos/asf/lucene/java/site/docs /www/lucene.apache.org/java/docs /usr/local/bin/svn export --force http://svn.apache.org/repos/asf/lucene/java/trunk/docs /www/lucene.apache.org/java/docs/nightly

I can add an scp from the nightly build if you just tell me where the files exist relative to the lucene trunk (i.e. build/...) and where you want me to put them on the website.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Grant, how are the nightly javadocs copied to the live site? Are the site/nightly/Hudson things documented anywhere?

A different idea: Is CGI allowed on lucene.apache.org? If so, the HTML view of CHANGES.txt could be dynamically generated for every request via a URL something like:

http://lucene.apache.org/docs/changes/changes2html.pl

The perl script would then need to have a hard coded path to CHANGES.txt.

It doesn't take the script long to execute, so this is feasible if traffic is low enough, and I'm guessing that this is so.

[asfimport]

Grant Ingersoll (@gsingers) (migrated from JIRA)

The nightly javadocs are on the Hudson server. They are not copied to people. We could just make them an artifact that gets stored by Hudson.

[asfimport]

Doron Cohen (migrated from JIRA)

I think there's an advantage in the build process generating this file. I will commit shortly this patch when 'nightly' target generates the html file under build/docs/changaes. From there we can copy it to the site. (more details later).

[asfimport]

Doron Cohen (migrated from JIRA)

I committed (part 1) and so target 'nightly' creates the HTML file (and copies the style files) under build/docs/changes/.

Grant, could you set the script to scp it under /www/lucene.apache.org/java/docs ?

So now /www/lucene.apache.org/java/docs/changes will contain the html file.
(or maybe /www/lucene.apache.org/java/changes is more adequate?)

[asfimport]

Doug Cutting (@cutting) (migrated from JIRA)

I worry that the nightly build documentation is too prominent. A search for "indexwriter api" on Google or Yahoo! returns nightly documentation before released documentation.

Nightly artifacts are for developers only, not the general public. Folks must be able to easily find released documentation. Nightly artifacts should be findable, but not prominent. I think having a single link to hudson on a developer oriented page of the website, and making the nightly docs findable through hudson only, is plenty. Any more and it seems like you're trying to promote the nightly outside the developer community, which is wrong.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Doug, do your concerns about nightly build documentation extend to the Changes.html generation being discussed in this issue?

I believe that Doron's intent is to link to the HTML version of the non-released trunk version of CHANGES.txt from the main web page (please correct me if I'm wrong, Doron). Do you object to this, Doug?

I ask because I think a case could be made for providing a window for users onto the nature of the changes that will be in the next release.

[asfimport]

Doug Cutting (@cutting) (migrated from JIRA)

Sure, it makes sense to make changes.txt relatively easy to find, like the bug database, so that folks can get an idea of what to expect in upcoming releases. Both should be part of the project's developer documentation, not the product documentation. I was mostly responding to the remarks above about copying the nightly docs to the public website, the motivation for which I don't fully follow.

[asfimport]

Doron Cohen (migrated from JIRA)

I worry that the nightly build documentation is too prominent. A search for "indexwriter api" on Google or Yahoo! returns nightly documentation before released documentation.

Yes.. just tried that.

... trying to promote the nightly outside the developer community, which is wrong.

I agree. Can this be avoided by robots,txt on hudson?

I believe that Doron's intent is to link to the HTML version of the non-released trunk version of CHANGES.txt from the main web page (please correct me if I'm wrong, Doron)

Well I'm not sure where they should be copied to - just want to make the changes accessible. Perhaps there's no need to copy the changes folder at all (see below).

Sure, it makes sense to make changes.txt relatively easy to find, like the bug database, so that folks can get an idea of what to expect in upcoming releases. Both should be part of the project's developer documentation, not the product documentation.

CHANGES.txt is already accessible from "Lucene News" in the main site. I think it makes sense to generate (once) Changes.html for 2.3 and add it to 2.3 branch and modify the main site to link to that file. Also, I think the 2.3 branch docs should link to this file. (I was planning to get to discuss this later.)

I was mostly responding to the remarks above about copying the nightly docs to the public website, the motivation for which I don't fully follow.

Only the changes directory is to be copied, Nightly documentation is copied already today from hudson to l.a.o/java/docs/nightly. Doug, are you concerned with current practice, or just with additionally copying the changes folder? if the latter. I think it is possible to link/redirect to Changes.html that is generated by the nightly build on hudson, the same as seems to be done for the javadocs links?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

In addition to the nightly HTML view of CHANGES.txt in the developer documentation, I think there should be a static HTML view of CHANGES.txt produced with every release; this per-release HTML view should be available from the product documentation for the corresponding release.

[asfimport]

Doron Cohen (migrated from JIRA)

{qoute} In addition to the nightly HTML view of CHANGES.txt in the developer documentation, I think there should be a static HTML view of CHANGES.txt produced with every release; this per-release HTML view should be available from the product documentation for the corresponding release. {quote} +1 : that's what I meant above (for 2.3)

[asfimport]

Doug Cutting (@cutting) (migrated from JIRA)

Maybe we can make the links to the nightly docs rel=nofollow?

[asfimport]

Doron Cohen (migrated from JIRA)

According to http://en.wikipedia.org/wiki/Nofollow#Interpretation_by_the_individual_search_engines will help with some search engines.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Perhaps the nightly docs thread should be moved to a new JIRA issue?

[asfimport]

Mike Klaas (migrated from JIRA)

If you just want to exclude them from search engines, tweak robots.txt or add the <meta name="noindex"> directive. nofollow should affect ranking, but possibly not existence, in the index.

[asfimport]

Doron Cohen (migrated from JIRA)

I created #2242 for the nightly docs.

[asfimport]

Doron Cohen (migrated from JIRA)

Last nightly build created the changes html directory

changes-to-html:
    [mkdir] Created dir: /export/home/hudson/hudson-slave/workspace/Lucene-trunk/trunk/build/docs/changes
     [copy] Copying 2 files to /export/home/hudson/hudson-slave/workspace/Lucene-trunk/trunk/build/docs/changes

What's the trick to link/browse the files there?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

If I browse to http://hudson.zones.apache.org/hudson/job/Lucene-trunk/changes/ , or anything in that directory, including Changes.html, I see a Hudson page dedicated to per-nightly-build commits. Nice page :). I'm guessing what's going on is a namespace issue: on the hudson server, anything you put into Lucene-trunk/changes/ is unlinkable-to, because that directory is dedicated to the Hudson Changes page.

Fixing this may be as simple as changing the name of the target directory, maybe to official-changes/ or something like that.

[asfimport]

Doron Cohen (migrated from JIRA)

I suspected something like this but wasn't sure... Ok I'll rename the directory and then we'll see.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Wait! I found it: http://hudson.zones.apache.org/hudson/job/Lucene-trunk/ws/trunk/build/docs/changes/Changes.html

[asfimport]

Nigel Daley (migrated from JIRA)

I suggest you save the Changes.html as one of the build "artifacts" (just like the tar.gz files are saved). Grant can add this file to the artifacts list in the Hudson configuration screen if you want this done.

[asfimport]

Doron Cohen (migrated from JIRA)

Nice spying work Steven :)

I am not familiar with Hudson configuration, but job/Lucene-trunk/ws/ sounds like a temporary work space, that might be erased during builds[?].

"artifacts" that are saved at the end of a successful build seem stable/proper. We need the entire changes directory - actually the 3 files in it - is "changes" an ok name for an "artifact" directory?

[asfimport]

Nigel Daley (migrated from JIRA)

job/Lucene-trunk/ws/ sounds like a temporary work space, that might be erased during builds

Yup, that's exactly what it is.

I've updated Lucene-trunk build to grab trunk/build/docs/changes/* at the end of the build and save them as artifacts.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Okay - it's available now at: http://hudson.zones.apache.org/hudson/view/Lucene/job/Lucene-trunk/lastSuccessfulBuild/artifact/trunk/build/docs/changes/Changes.html

Wow, that's a looooooong URL. Can we shorten that at all? E.g.:

http://hudson.zones.apache.org/hudson/view/Lucene/job/Lucene-trunk/lastSuccessfulBuild/artifact/changes/Changes.html

[asfimport]

Doron Cohen (migrated from JIRA)

Ok great!

Now we can link to this page, and developer-resources seems the right place.

But developer-resuorces.xml was deleted from svn in r611094 http://svn.apache.org/viewvc?view=rev&revision=611094

I think it was deleted by mistake?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

But developer-resuorces.xml was deleted from svn in r611094

That's strange - this deletion was committed on 11/Jan/08, but the corresponding HTML page on the site says it was published on 24/Jan/08 ???

I can see no other way for that HTML page to be generated, either.

Michael B., do you know what's going on here? You appear to be the deleter.

[asfimport]

Chris M. Hostetter (@hossman) (migrated from JIRA)

note the commit mesg: "Separate project's web site from version-specific documentation." ... i guess Mikchael did it as two separate commits. this is the new home of "the site" where docs that are not version specific live...

http://svn.apache.org/viewvc/lucene/java/site/

(ie: it's not part of the trunk, and won't be branched on every release)

[asfimport]

Doron Cohen (migrated from JIRA)

Thanks Hoss, got it now.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Excellent, the link from the Developer Resources page now works!

Doron, I noticed that when you initially committed this, you added an entry to CHANGES.txt, in the "Documentation" section, but it is no longer there.

[asfimport]

Doron Cohen (migrated from JIRA)

Yes, I was kinda not sure what to do next - close this (and update changes.txt) and that's it, or first migrate more links from linking to the raw changes.txt to link to the fancier changes.html.

The announcement of 2.3 in the main page which now links to 2.3/changes.txt is one candidate. The main page of 2.3 docs is another good place for such a link. Generating the file once is easy enough, and can be done in the future as part of closing the release.

But where do you link to? I.e. where to put the generated file?

One option is to add it to svn (when the release is closed), and link to it e.g. like this: http://svn.apache.org/viewvc/lucene/java/branches/lucene_2_3/Changes.html?view=co But then future changes to the branch are likely to forget updating this file...

Steven, what do you think of this?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

I think the simplest thing to do would be to put the Changes.html (and .css files) for the most recent release in the non-versioned /site/ tree:

http://svn.apache.org/repos/asf/lucene/java/site/docs/changes/

It would be updated there as part of the release process. And it could be linked to from everywhere that now links to CHANGES.txt.

In addition, the per-release documentation (maybe from the Overview page?) could link to a version of Changes.html that only includes changes up to the release in question. AFAICT, the per-release documentation is created as part of the release process.

That leaves the question of whether to include HTML versions of CHANGES.txt for previous releases - my vote is to do so.

That makes 4 different classes of location for Changes.html:

1. Current trunk view, linked from developers' documentation. (done)
2. Latest release view, linked from the non-versioned site.
3. Per-release view for releases going forward - contains changes only up to the release.
4. Per-release view for previous releases - contains changes only up to the release.

[asfimport]

Doron Cohen (migrated from JIRA)

So there will be: http://svn.apache.org/repos/asf/lucene/java/site/docs/changes/Changes_2_1_0.html http://svn.apache.org/repos/asf/lucene/java/site/docs/changes/Changes_2_2_0.html http://svn.apache.org/repos/asf/lucene/java/site/docs/changes/Changes_2_3_0.html

These will be created once, now, for previous releases, and for future releases as part of the (to be updated) release steps.

I like this except for one thing: what it means for fixing a release branch. Today fixing a release branch requires to also update CHANGES.txt. From now on it would require to generate Changes.html and check it in. My concern is that people would forget to update this file, and we will have a nice html view of changes but it would not be up-to-date.

Up to today I saw branch updates only as part of last minute fixes of the to-become-a-new- release. But it is not impossible that critical fixes will be merged in the future into an existing releases.

"Release To Do" can be extended with the required steps for fixing/updating a release branch. Being a rare practice this might be enough...?

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

Hi Doron,

I've created a new issue for some improvements to the Changes.html formatting, at #2333. When you get a chance, would you please take a look?

[asfimport]

Doron Cohen (migrated from JIRA)

I'll do it over the weekend.

[asfimport]

Doron Cohen (migrated from JIRA)

Thanks Steve! I'll create a another issue for making Changes.html visible also to users for closed releases.

[asfimport]

Steven Rowe (@sarowe) (migrated from JIRA)

The follow-on issue