Add an ant task to verify download sources still exist

[GoogleCodeExporter]

Suggestion to help with repository maintenance and make it easy to detect
resources that no longer are available at the URLs specified in their
packager.xml files:

Create a new ant task in build.xml that runs an XSL transform over
modules.xml (and which processes all packager.xml files) to generate a new
ant build file that verifies the existence of all of the archives specified
in <resource> "url" attributes and <artifact> maven resource tags. You can
do this with a timestamp hack like this:

  <touch file="timestamp"/>
  <get dest="timestamp" usetimestamp="true" src="http://..."/>

This will not actually download the resource (unless the resource has a
timestamp in the future) but will verify that it exists.

Original issue reported on code.google.com by archie.c...@gmail.com on 1 Mar 2010 at 7:20

[GoogleCodeExporter]

You could use
<condition property="res1-exists">
  <http url="http://..."/>
</condition>
if want a property to be set, too.

Original comment by fifteenk...@gmail.com on 1 Apr 2012 at 3:58

[GoogleCodeExporter]

I would volunteer on this, but my knowledge of XSLT is very limited.
I made an Ant target that generates a script to verify download URLs, but i 
don't know how to transform each packager.xml into a file with content like 
this:

    <get-and-print packagerfile="src/module/com.icu/1.3.1/packager.xml">
      <url url="http://www.icu-project.org/repos/icu/icu4j/tags/release-49-1/x1.jar"/>
      <url url="http://www.icu-project.org/repos/icu/icu4j/tags/release-49-1/x1-source.zip"/>
      <url url="..." />
    </get-and-print>

Original comment by fifteenk...@gmail.com on 18 May 2012 at 1:57

[GoogleCodeExporter]

Here's the first try. It still does not handle resources properly, wich have to 
downloaded manually.

Cheers,
 Martin

Original comment by fifteenk...@gmail.com on 20 May 2012 at 6:46

Attachments:

• verify-url.patch

[GoogleCodeExporter]

Thanks! I added your patch in r2112. We can add more to it later as needed, etc.

Original comment by archie.c...@gmail.com on 24 Jun 2012 at 8:56

• Changed state: Fixed

[GoogleCodeExporter]

Sorry, I am surprised that my patch was incorporated 'as is'. It is just a 
proof of concept.
At least the dependency in the imported build-urlcheck.xml may raise a 
maintenance issue: The check target depends on a target defined in the 
importing file.

Original comment by fifteenk...@gmail.com on 10 Jul 2012 at 7:13

[GoogleCodeExporter]

That's OK, it's not hurting anything by being there. I checked it in so anyone 
could continue to play with if desired.

Original comment by archie.c...@gmail.com on 10 Jul 2012 at 8:09

[GoogleCodeExporter]

Attached is a patch that also verifies the homepage-URLs.
Resources that must be downloaded manually, are skipped now.

To find invalid URL, run
ant check | grep -i "Can't get http"

Original comment by fifteenk...@gmail.com on 11 Nov 2012 at 4:42

Attachments:

• verify-homepage.patch

[GoogleCodeExporter]

Thanks, added in r2361.

Original comment by archie.c...@gmail.com on 12 Nov 2012 at 4:43

[GoogleCodeExporter]

Mostly a complete rewrite this time.

Designed to help eliminate invalid URLs.
- Parallel execution reduces time to run from 1 hour to ~ 5 minutes.
- Forked JVMs avoid OOM exception when running the script.
- Caches module revisions with no error to speed up subsequent runs.
- Produces reports with no clutter from Ant: REPORT-OK.txt & 
REPORT-problems.txt.
- Uses URLs found in src/modules instead of repo/modules.

Archie, I put verify-url-lib.xml directly under src. If you find a better 
location for the file, I will fix that.

Happy maintaining,
 Martin

Original comment by fifteenk...@gmail.com on 26 Nov 2012 at 10:08

Attachments:

• verify-url.patch

[GoogleCodeExporter]

Patch committed in r2380. I also relocated the support files into src/xml.

Original comment by archie.c...@gmail.com on 27 Nov 2012 at 3:43

[GoogleCodeExporter]

Patch changes
- Also verifies license URLs (BTW: found some without a protocol: 
url="LICENSE.txt")
- New target prep-script creates script that edits files in src/modules for 
permanently moved URLs.

Original comment by fifteenk...@gmail.com on 29 Nov 2012 at 9:33

Attachments:

• verify-url2.patch

[GoogleCodeExporter]

Thanks, patch applied in r2400.

Original comment by archie.c...@gmail.com on 30 Nov 2012 at 2:33

[GoogleCodeExporter]

Made generated script that edits files for permanently moved URLs more 
foolproof: Updated URLs are verified before the ivy.xml and packager.xml file 
get changed.

Original comment by fifteenk...@gmail.com on 1 Dec 2012 at 3:50

Attachments:

• verify-url3.patch

[GoogleCodeExporter]

Thanks, added in r2408.

Original comment by archie.c...@gmail.com on 3 Dec 2012 at 4:35

[GoogleCodeExporter]

Patch assumes a 'moved to' HTTP response is caused by load balancing/mirroring 
mechanisms.
Validation report get much shorter with it.

Original comment by fifteenk...@gmail.com on 5 Dec 2012 at 9:00

Attachments:

• verify-url-moved.patch

[GoogleCodeExporter]

Thanks, patch applied in r2417.

Original comment by archie.c...@gmail.com on 5 Dec 2012 at 9:54