defocal pair tools

[leginonbot]

---

Author Name: Scott Stagg (Scott Stagg) Original Redmine Issue: 966, https://emg.nysbc.org/redmine/issues/966 Original Date: 2010-10-25

---

I have been doing defocal pairs lately, and there are a few features that are missing that I would like to add as little convenience tools. First, I would like to be able to extend image assessment from an image to its sibling (i.e. ef to en). I would also like to do the same with particle picks. I know that makestack will do the latter, but I want to be able to see the picks on the web imageviewers. Finally, I would like to be able to make stacks of focal pairs where there is a one-to-one correspondence between the particles in the different stacks. I am looking for suggestions as to the appropriate approach for writing these scripts. Should there be separate scripts for each task? Should they be rolled into other scripts as features? Thoughts? Also, I don't have the time to add webpage launchers for these scripts. I would need someone else to do that stuff. If that's not kosher, I can just do it locally. Please add any watchers who would be interested in this discussion.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2010-10-26T14:53:56Z

---

Let's break them up:

1. Extend image assessment to its sibling. Do you mean that if you reject en, then ef is marked rejected? If this is done in manualpicker.py I recommend doing it in the same script then the web interface will be easier.

2. Paired particle picks. I've noticed that alignDefocalPairs.py stands alone and makestack2.py need its database entry to find the corresponding position. I think this one should be done at the webviewer level by using that database entry, not to create a whole set of new particle data records.

It is o.k. to do the python part and assign the web part to others as long as instruction on what error checking is needed and how to link the two are clear, very clear. You should also assume the responsibility of testing it.

Could you open new issue for each and assign the first to yourself to do the python part and once that is completed it can be assigned to Eric or Amber to do the web part.

For the second issue, Eric has agreed to make a web interface for alignDefocalPairs.py and the particle mapping. You should give some instruction about the former. I will help him with finding the components I learned from recent experience for the latter.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2010-10-27T13:55:47Z

---

OK, I will add new issues, but first I want to discuss a little more. I'm not sure what you mean by #1. In my example, I used the web viewer to reject ef images. Then I wanted to make a stack from en images but exclude the images that had their siblings rejected. I wrote the attached prototype script to extend the assessments to the siblings and record that in the db, but it has some problems. For instance, since the en images haven't been picked, the counts for the assesssumary.php page are completely off. Maybe extending the assessment is something that should be an option in makestack2.py? Then the problem becomes how to have it make sense to a user.

I agree that having the webviewer display the picks instead of having a whole other set of picks is a better solution. However, it doesn't solve the problem of having a one to one correspondence of picks in an en stack and an ef stack. One potential problem is that an en image might be rejected by CTF while its ef sibling is included. Another is that a particular particle that is included in ef image might be out of bounds in an en image because of some slight drift. Would a better solution be to use makestack to create en and ef stacks separately, then have a separate script query the db, find common pick entries, then make two new stacks with a one to one correspondence between particles? Thoughts? Neil and Gabe, do you want to weigh in?

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2010-10-28T13:48:54Z

---

Scott,

You are right about the correspondance issue in the stacks. I think we will need to have both display of sibling picks in the imageviewer. makestack can make either sibling already but a table for paired particles in stacks is needed if query through ef picks and defocal pair shift is too inefficient.

Neil, how does RCT deal with correspondence issue in the stack of tilted and untilted particles now? I imagine the problem is similar.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2010-10-28T14:29:06Z

---

I don't think you'd need a new table for paired particles. If you made a stack of en particles and another stack of ef particles all from the same pick run, then the corresponding particles will have the same pick parent in the database. All you would need to do then is find the common particles and make two new stacks from the original ones where the new stacks will have one to one correspondence. Does that make sense?

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-10-28T16:39:26Z

---

Scott, for RCT I have to insert a particle for both images, because the tilt transform is not perfect due to distortions in the grid height, for defocal pairs I assume the x-y shift and zoom transform is all you need to perfectly reproduce the defocal particle pair, so two entries is not necessary.

On another note, I set up a "defocal pair" system for Anna-Clare to test her direct detector data. She has high dose and low does images rather than defocal pairs, but she is using the same system. To this end, Appion's FREALIGN allows you to use one stack for alignment (high dose or defocus) and a second stack for backprojection (low dose or defocus).

What I have her do is create two stacks from a single particle picking run, one from each defocus. I know that works fine. I am not exactly sure what you are trying to implement, so I cannot comment further.

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2010-10-28T17:58:36Z

---

Scott,

Do you mean then you want to make two stacks one en one ef but to make sure that the 1256th particle in en stack to be picked from the equivalent position in the 1256th particle in ef stack? In other words to throw out particles in ef stack that do not correspond? I guess it is either a substack job or a specialized makestack. The latter may be better because we still have an unsolved problem of substack that could not take a very long list of particle exclusion.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-10-28T19:08:32Z

---

Yes, that was the problem. We had no good way to make sure the particle were perfectly in sync for each stack that was then required for FREALIGN. Anna-Clare situation was simple, because the shift was always (0,0) it is the same image, so no additional particles were thrown out in one preset and not the other. But that is the major problem.

In RCT, the database solves the problem for me. I do not care what order the particles are in the stack I just look up the corresponding particle in the database. Something to think about.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2010-10-28T19:31:52Z

---

Neil, I'm a little lost. I don't understand why your solution for Anna-Clare would work. There is no guarantee that the particles would be the same in the en or ef stacks.

Anchi, yes, that is what I'm suggesting. I think adding an option in makestack might be a little unwieldy. My suggestion would technically produce substacks, but they aren't necessarily required. They could be separate independent stacks, I think.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-10-29T17:54:24Z

---

Scott, sorry if I was confusing. You are right there is NO guarantee the particles will match up.

In her case, the images are exactly same location it is just the number of frames going into image that make them different. So, in that case, we usually had no problems, because the particle picking run was the same and the assessment IS INHERITED between defocal pairs in makestack (it fails for speed and clarity reasons in assess_summary.php). So, in her case, they were matched up. The only time we had problem is when the defocal shift calculation failed to find the peak and inserted an almost random shift, but that was fixed.

We need a mechanism to guarantee that they match up. We could have makestack produce two stacks, one from each preset, simultaneously and force matching.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-10-29T17:58:23Z

---

Another option that would be easier to implement, but its major drawback is that it creates extra data.

If you look at my program @tiltStackSync.py@ it is designed to take two existing stacks from the same tiltpicker run and produce two new stacks with the particles exactly sync'd. Gabe and Ed have been using this to export RCT stacks outside of the pipeline for custom SPIDER processing.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2010-10-29T18:09:06Z

---

Are you saying that makestack already will already extend assessment between defocal pairs? In other words with the current makestack, if I reject an ef image its corresponding en image would be skipped in a makestack run on en images?

What you are describing with tiltStackSync is exactly what I have been suggesting. I think it might be worth using the extra disk space to have a program like this for defocal pairs instead of making yet another option for makestack. Makestack is so complicated already.

[leginonbot]

---

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-10-29T19:00:15Z

---

Sorry, no you have to assess the images for each preset. makestack2 uses the appionLoop2 image rejector, which calls the apDatabase.getImgCompleteStatus(), which has no defocal pairs component. There is a function, apDatabase.getSiblingImgAssessmentStatus() that does this, but probably needs updating.

In our case, we uploaded images, so they were all good, but if you only pick particles on the correctly assessed images you should be good, but this is definitely not a fool-proof system.

[leginonbot]

---

Original Redmine Comment Author Name: Scott Stagg (Scott Stagg) Original Date: 2010-10-29T19:15:14Z

---

OK. Based on all the input. I am going to do two things. 1) I will make a new option for makestack called --ea (for extend assessment) which will use the sibling's assessment data to determine whether to skip an image

2) I will make a new program called focalPairSync.py that will take an en stack and an ef stack with the same picking run and create two new stacks with a one to one correspondence between particles in the two stacks.

I will make new issues for each of these items. If anyone has any comments or problems with that, please let me know now.

P.S. Redmine is awesome for facilitating this discussion. I feel so much more in the loop now!

[leginonbot]

---

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2010-10-29T19:47:34Z

---

Sounds like a good plan. Looking forward to the new feature.