Open leginonbot opened 5 months ago
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-08-27T17:55:18Z
r18540 is done in myami-dbcopy branch. This feature is not possible in the main branch.
Sargis, if you want to look at the code, you should only need the sinedon subpackage.
I will upload an example on how I use it later.
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-08-29T01:06:17Z
r18544 adds the script for archiving a project by project id.
To use:
python archive_project.py 1
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2014-08-29T17:32:26Z
Thank you Anchi. I tried this on one of the projects and I got the following error:
In [34]: %run /home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py 231
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
/usr/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
176 else:
177 filename = fname
--> 178 __builtin__.execfile(filename, *where)
/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py in <module>()
708
709 checkSinedon()
--> 710 archiveProject(projectid)
/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py in archiveProject(projectid)
688 p = projectdata.projects().direct_query(projectid)
689 source_sessions = projectdata.projectexperiments(project=p).query()
--> 690 session_names = map((lambda x:x['session']['name']),source_sessions)
691 session_names.reverse() #oldest first
692 for session_name in session_names:
/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py in <lambda>(x)
688 p = projectdata.projects().direct_query(projectid)
689 source_sessions = projectdata.projectexperiments(project=p).query()
--> 690 session_names = map((lambda x:x['session']['name']),source_sessions)
691 session_names.reverse() #oldest first
692 for session_name in session_names:
TypeError: 'NoneType' object has no attribute '__getitem__'
Working on debuging this...
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-09-02T22:50:56Z
Sargis
r18552 fixed the bug that caused the error and also improved the speed a lot by using sinedon timelimit query rather than timestamp comparison in python which had to compare all results since python iteration is slow.
The cause of the bug was that some early imported calibrations have no session field (default to None).
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2014-09-03T14:18:08Z
Thank you Anchi. This time it did a good progress and I'm now getting this result:
[sargis@dewey ~]$ python /home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py 231
['09may25a', '09may25b', '09may25c', '09jul11a']
****Session 09may25a ****
Importing session....
Importing instrument....
Importing calibrations....
Importing C2ApertureSize....
number of images in the session = 278
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
imported 278 images
Importing image ddinfo....
Importing image stats....
Importing queuing....
Importing mosaic tiles....
Importing dequeuing....
Importing drift....
Importing focus results....
importing TransformManager Settings....
Traceback (most recent call last):
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 742, in <module>
archiveProject(projectid)
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 726, in archiveProject
app.run()
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 710, in run
self.runStep3()
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 696, in runStep3
self.importSettings()
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 623, in importSettings
self.importSettingsByClassAndAlias(allalias)
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 590, in importSettingsByClassAndAlias
results = self.researchSettings(settingsname,name=node_name)
File "/home/sargis/ami-workspace/myami-trunk/dbschema/archive_project.py", line 188, in researchSettings
r2 = q.query()
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/data.py", line 429, in query
results = db.query(self, **kwargs)
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/dbdatakeeper.py", line 99, in query
result = self._query(*args, **kwargs)
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/dbdatakeeper.py", line 118, in _query
result = self.dbd.multipleQueries(queryinfo, readimages=readimages)
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/sqldict.py", line 250, in multipleQueries
return _multipleQueries(self.db, queryinfo, readimages)
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/sqldict.py", line 525, in __init__
self.queries = setQueries(queryinfo)
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/sqldict.py", line 983, in setQueries
query = queryFormatOptimized(queryinfo,value['alias'])
File "/home/sargis/ami-workspace/myami-dbcopy/sinedon/sqldict.py", line 1020, in queryFormatOptimized
joinTable = queryinfo[id]
KeyError: 41493216
Will look into this...
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-09-09T17:46:54Z
r18561 fix this problem. Sinedon thinks "source_session" is in the destination_dbname even though the original db is source_dbname. Work around that by redo the query for source_session in get.
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-10-16T20:53:26Z
r18645, r18646, r 18647 creates a working pipeline
These include:
DEF_id
.Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2014-12-12T01:53:32Z
r18698 and r18799 gives more options to leginon archiving
Author Name: Anchi Cheng (@anchi2c) Original Redmine Issue: 2915, https://emg.nysbc.org/redmine/issues/2915 Original Date: 2014-08-27 Original Assignee: Sargis Dallakyan
leginondata is getting too big. The queies are slow.