harvest.pl script production problems

spikelynch / CoataGlue

ANDS DC18 common codebase for data crosswalk and publication

Other

0 stars 0 forks source link

harvest.pl script production problems #5

Closed spikelynch closed 11 years ago

spikelynch commented 11 years ago

There are some niggles to work out of harvest.pl in production - it was designed for a testing model where either all of the test sets were new, or they were all old. What we need:

a way to specify a start index for the global IDs
a way to explicitly re-harvest a dataset which has already been seen before
utilities to view the Store (as it's a Storable bin, not a text file)

spikelynch commented 11 years ago

I've got the script to good-enough status - it can dump out the contents of the store.

Explicit re-harvesting will have to wait till later.

spikelynch commented 11 years ago

Explicit re-harvesting is now working. It's not perfect but I'm closing this issue and will open another one with some details about how it should be upgraded.