Currently the submit script will work on a file containing a list of archive files.
The general idea is:
extract all the PDB files from the archives into a given directory
process all the PDB files
This means that there is heavy network IO at the start of the job. It would be better to spread the network load out at to random points throughout the job - so..
for each archive file:
extract all PDB files
process all PDB files
One extra bonus is that this can provide the opportunity to record milestones that a given archive file has been processed correctly (and can potentially be ignored on future runs).
Currently the submit script will work on a file containing a list of archive files.
The general idea is:
This means that there is heavy network IO at the start of the job. It would be better to spread the network load out at to random points throughout the job - so..
for each archive file:
One extra bonus is that this can provide the opportunity to record milestones that a given archive file has been processed correctly (and can potentially be ignored on future runs).