document how to test seed and crawl accessioning

[ndushay]

The current documents on consul are lacking for developers. There is some existing documentation in consul:

• https://consul.stanford.edu/display/WARC/Web+Archiving+Development+Documentation
  • https://consul.stanford.edu/display/WARC/WAS+Registrar+-+Registering+Crawl+objects+from+file
• https://consul.stanford.edu/display/WARC/Crawl+Object+Accessioning+Walkthrough

but this documentation is not targeted towards developers testing code/services.

We need additional information so we can be more confident we know how to smoke test the was code/services. It will also help us write the missing specs.

We need to know things like:

• was-registrar kicks off the was robots
• was-robots preassembly WF kick off common accessioning WF, and then common accessioning WF kicks off the was-robots dissemination WF.
• how do you put a new (test) seed into was-registrar and kick off robots?
• how do you set up a new (test) crawl and kick of was-registrar from rake task?
• how do you monitor where it is in was-registrar processing?
• how do you monitor where it is in robot steps?
• how do you vet that the object went through all the steps?
• how do you vet that the final result is ok?
• how does a seed or crawl show up in argo?
• when does a seed or crawl go beyond argo?

Checklist might be:

• [ ] how to test seed accessioning

  • [ ] how to put seeds in was-registrar
  • [ ] what should be in tsv file?
    • [ ] sourceid must be unique
  • [ ] how to register the seed in was-registrar
  • [ ] now it's up to the robots ... then what?
  • [ ] what should finished seed look like ... in argo?
  • [ ] can/should we 'publish' seed to see if it's ok in SearchWorks or other discovery environment beyond argo?
  • [ ] what should happen in thumbnail service?
  • [ ] how to check seed in thumbnail service
  • [ ] how to check for errors processing seed in thumbnail service

• [ ] how to test crawl accessioning

  • [ ] how to prep crawls for accessioning
    • [ ] what preconditions must exist (crawl completed and stored ... where? how?)
    • [ ] what should be in tsv file?
  • [x] how to use was registrar rake task to register crawl objects (see Consul)
    • [x] where should tsv file be?
    • [x] how to know if registration worked?
    • [x] how to check for errors in registration
  • [ ] now it's up to the robots ... then what?
  • [ ] what should finished crawl look like ... in argo?
  • [ ] can/should we 'publish' crawl to see if it's ok in SearchWorks or other discovery environment beyond argo?
  • [ ] what should happen in swap? (NOTE: with crawls, swap updates only after accessioned and disseminated)
  • [ ] how to check crawl in swap
  • [ ] how to check for errors processing crawl in swap

[drh-stanford]

i was successful in testing a crawl object in stage with the new multilevel indexing. I made some edits to the current documentation found here: https://consul.stanford.edu/display/WARC/WAS+Registrar+-+Registering+Crawl+objects+from+file