search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

bake appion processing recipes #2836

Open leginonbot opened 8 months ago

leginonbot commented 8 months ago

Author Name: Gabriel Lander (@gclander) Original Redmine Issue: 2836, https://emg.nysbc.org/redmine/issues/2836 Original Date: 2014-07-14 Original Assignee: Neil Voss

Add function to save a stack or aligned stack as a recipe, enabling the same processing steps to be run for future experiments.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Gabriel Lander (@gclander) Original Date: 2014-07-16T21:33:41Z

First step toward saving recipes. Stacks and aligned stacks can be saved and removed from the database, the recipes show up on the main page if they exist. TO DO: update database schema: myami/myamiweb/xml/appion_extra.xml myami/dbschema/schema-new.py myami/dbschema/updatelib.py

Have to figure out how this will be launched & executed

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Clint Potter (Clint Potter) Original Date: 2014-08-27T15:37:55Z

8/27 Appion conference call. Need to generate parent command that works on multiprocessors. Need to know how many processors to ask for. Currently not tracking number of processors. Try with defaults for now. New table stores recipes.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Amber Herold (Amber Herold) Original Date: 2014-08-29T02:33:02Z

Gabe, The runJob.py command is being saved to a file in the run directory. You may be able to parse this file to find the number of nodes and processors requested in many cases, assuming the run directory still exists. Of course it would be better to have this info in the DB. Perhaps as another column in the existing run data table.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Clint Potter (Clint Potter) Original Date: 2014-11-19T16:29:47Z

Discussed during Appion conference call. Gabe added a button to start. Gabe doesn't have much time.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2016-02-08T16:56:33Z

I am going to take a look at this. Has anything been started on it. I assume it will be using the Script tables in the Appion database to generate commands or is it solely based on the runJob log files? I will have to look at new tables for this purpose.

The main problem I do not know the answer to... is how do we launch a new command after the first one completes. Pick particles then make stack (1). I could have the user click through the steps as they complete to start, but then it is not automated. (2) Or I could put multiple commands into the same job script, but different commands have different resource needs. Perhaps the new job submission setup can allow for delayed jobs or something. Any thoughts would be appreciated.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Gabriel Lander (@gclander) Original Date: 2016-02-09T02:38:42Z

I spent a lot of time trying to figure what to do on this front. I think my plan was to generate all the necessary individual scripts, then have a master script that launched them. The master script would wait for them to complete before launching subsequent ones. Using this approach, the master script could simultaneously launch particle picking & CTF estimation, but wait for them both to finish before starting stack creation.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Bridget Carragher (@bcarr-czi) Original Date: 2016-02-09T18:49:36Z

Could we add a trigger (either as a file that appears) or a database entry so that the next up scripts can query for completion and then start once they get the go ahead. They could query on a longish time scale....