was-downloader: document

[ndushay]

See:

• https://consul.stanford.edu/display/WARC/Download+ARCs+from+CDL+WAS
• https://consul.stanford.edu/display/WARC/WAS+Code+Projects
• https://consul.stanford.edu/display/WARC/Web+Archiving+Machines+and+Monitoring

If this is a separate entity from hrwa_manager:

If it is code, minimum:

• [ ] it should be in github
• [ ] it should have devops docs of at least a README that has a couple sentences about what it is, where the git repo is and what VMs it is on. If it appears to be deployed manually, say that. Say how it relatest to hrwa-manager

If it is not code, but a concept about some code:

• [ ] add a devops doc README-was-downloader.md or something explaining what you know and how it relates to hrwa_manager and other stuff.

[ndushay]

another question for @nullhandle : are we using fog or a python library for was-downloader? consul has docs for a ruby script using fog, but the puppet manifest includes the python-pip package.

[ndushay]

ait_download

Short term:

• leave box alone. It is useful for archive-it downloads, which compute checksums and shouldn't be competing with robots for resources.
  • confirm that hrwa_manager is doing checksums.

Long term:

• reconfigure the code in /web-archiving-stage/apps into github and use the usual approach (in /opt/app/xxx)

cdl_download , lockssssssss export

want to keep this code / data / logs somewhere but it is deprecated