This is likely to fail a lot. My current thoughts are to have the pipeline ignore any errors when doing the layout and just run it as many times as it takes to get all the data in. If layout errors are ignored it should update the database (pipeline_tracking and rnc_secondary_structurelayout*) for only the chunks that finished. This would give us a partial update we can work with.
As part of this the pipeline will write the files to disk for backup. We should write to LTS (which I need to find) for storage.
This is likely to fail a lot. My current thoughts are to have the pipeline ignore any errors when doing the layout and just run it as many times as it takes to get all the data in. If layout errors are ignored it should update the database (pipeline_tracking and rnc_secondary_structurelayout*) for only the chunks that finished. This would give us a partial update we can work with.
As part of this the pipeline will write the files to disk for backup. We should write to LTS (which I need to find) for storage.