search

ePSIC-DLS / epsic_tools

Code for conversion and analysis of Merlin-Medipix data
GNU General Public License v3.0
6 stars 7 forks source link

Post conversion processing #15

Closed chrisallen80 closed 4 years ago

chrisallen80 commented 5 years ago

Code for running processing on 4DSTEM data on the cluster. Currently implemented are: ADF, COM and iDPC Have tested on one user data set with no errors.

TomSlater commented 5 years ago

I got rid of a couple of import statements in order for it to run for me, after which it seems to work.

If we're going to be using py4DSTEM regularly it might make sense to try to add it to the py3.7 environment, although I don't know whether they'll be happy to do this from a Github repo vs installing from conda or pip.

M0hsend commented 5 years ago

I got rid of a couple of import statements in order for it to run for me, after which it seems to work.

If we're going to be using py4DSTEM regularly it might make sense to try to add it to the py3.7 environment, although I don't know whether they'll be happy to do this from a Github repo vs installing from conda or pip.

I have raised this with Tom Schoonjan and his response was he'd rather wait for a PyPI release: https://jira.diamond.ac.uk/browse/SCI-8717

chrisallen80 commented 5 years ago

I've got batch processing working. use: qsub -t 1-4 batch_process_STEM.sh where 1-4 is the number of files to process simultaneously (in this case the first in the list to the forth in the list). We should be able to first scan the folder and return this number. I'm working on a .py file to do this then call qsub…….

chrisallen80 commented 5 years ago

This now works! From the folder containing process_cluster.py you can run module load python/3.7 python process_cluster.py beamline year session It points to the e02 folder on dls_sw where all the other code is stored. Everyone has read access to this folder, as long as we keep this up to date the only downloading of code required is the process_cluster.py file. Eventually we should have this in a module load epsic. In process_cluster.py there is a max_c variable which sets the maximum number of concurrent processes to run. I've set to 10, not sure what the upper limit should be. All data will be processed, just 10 at a time....

This functionality should be added to the mib2hdf conversion code.

chrisallen80 commented 4 years ago

Added toolbox, PtyRex and probesim folders. toolbox has generally userfull code snippets in - e.g. calculate probe overlap.

TomSlater commented 4 years ago

@M0hsend and @chrisallen80 , could you check the 3 conflicting files above and which parts you'd like to keep? Or, if you know if master is more up to date than the PR I can just resolve them all to versions in master.

chrisallen80 commented 4 years ago

@M0hsend , I think this is probably on you as it is the conversion code.

M0hsend commented 4 years ago

@M0hsend , I think this is probably on you as it is the conversion code.

I think this PR should not change the conversion code, if not directly linked to it. This has probably happened because your branch was not uptodate with master when you merged @chrisallen80 ? That said, this conversion code needs to change soon to call pyxem load_mib anyways. So if a bit hassle to roll back, we can merge now and resolve conversion issues in a separate PR.

TomSlater commented 4 years ago

Okay, I will keep all of the master conversion code when resolving the conflicts here.

TomSlater commented 4 years ago

Is this okay to be merged now or is there anything else that needs to be changed?