Hi Jody,
...
Anyways the pipeline is still running the samples on jasper. The way the pipeline was designed was to handle large samples, not many small samples. In short the way it is running is not very optimal. I'll bring this issue up with Kishori as it shouldn't be a large change and it is just a good improvement.
Kishori I'll make this a new issue on the MetaPathways github. Essentially jobs are submitted one database at a time and then it waits to collect the results. This is fine for large samples when there are enough samples to fill the queue, however, if there is not the daemon script may as well try to fill the queue with the jobs for the other databases. I understand that this might muck up the progress bar, but really that needs to be updated anyways as it is confusing for the most part.
Sorry for the delay, but this turned out to be a good case where MetaPathways is running slow (by not submitting enough jobs at once) for no good reason.
Hi Jody, ... Anyways the pipeline is still running the samples on jasper. The way the pipeline was designed was to handle large samples, not many small samples. In short the way it is running is not very optimal. I'll bring this issue up with Kishori as it shouldn't be a large change and it is just a good improvement.
Kishori I'll make this a new issue on the MetaPathways github. Essentially jobs are submitted one database at a time and then it waits to collect the results. This is fine for large samples when there are enough samples to fill the queue, however, if there is not the daemon script may as well try to fill the queue with the jobs for the other databases. I understand that this might muck up the progress bar, but really that needs to be updated anyways as it is confusing for the most part.
Sorry for the delay, but this turned out to be a good case where MetaPathways is running slow (by not submitting enough jobs at once) for no good reason.
Cheers, Niels