search

embl-cba / bd-cellview

5 stars 0 forks source link

Testing 15.10.20 #18

Open schraivo opened 3 years ago

schraivo commented 3 years ago

@tischi

I tested again and these are my observations:

I guess I will soon start to re-use json files to change parameters and paths, and only go into the fiji processing tool if I have a very new dataset for which I have to determine the thresholds first. Would it be possible to give the json a more meaningful name, e.g. with day/date as done for the jpeg folders? This will make the recycling more convenient.

Another question: I guess the logic for where the json file will be saved is to put it into the parent folder of the first table in the list of tables? I'm absolutely fine with, but just to double-check.

Thanks!! Daniel

tischi commented 3 years ago

I guess the logic for where the json file will be saved is to put it into the parent folder of the first table in the list of tables?

No, the json files go into the directories of the respective tables => you currently have to collect them from the different locations for the slurm processing.

Processing images from several tables works in principle, but we have the same issue as reported recently: only from the first table in the list of tables it processes 3003 images (instead of the number of images I set). Also, it only processes the images from the 1st table, and ignores the next few tables. But it does say that the job was finished, btw.

I saw that there were two jobs executed.

For the first one it was 3003 images and for the second one 100 images were processed.

Job 0

Images matching gate: 3003
Saving processed images to directory: /g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP/images-processed-2020-10-15-11-46
Images to be processed: 3003

Job 1

Images matching gate: 4800
Saving processed images to directory: /g/bdimsort/200807_Dewi_Nsp1/20200830_1648_HeLa RIEN NSP1 wt/images-processed-2020-10-15-11-46
Images to be processed: 100

Does this make sense?

schraivo commented 3 years ago

The "Job 0" should actually process 100 images from three tables each (that's what I've entered), but the json says that 3003 images should be processed. Also, the process stops after the first table is done and doesn't go to the second table.

This is the corresponding json file:

{ "tableFiles": [ { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1648_HeLa RIEN NSP1 wt/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1740_HeLa RIEN NSP1 mut1/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1755_HeLa RIEN NSP1 mut2/output/reduced_gated.csv" } ], "minimumFileSizeKiloBytes": 10.0, "maximumFileSizeKiloBytes": 100000.0, "whiteIndexString": "1", "minWhite": 0.0, "maxWhite": 0.17, "greenIndexString": "5", "minGreen": 0.0, "maxGreen": 0.1, "magentaIndexString": "3", "minMagenta": 0.08, "maxMagenta": 1.0, "viewingModality": "Processed Overlay And Individual Channels", "maxNumFiles": 100, "experimentDirectory": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP", "selectedTableFile": { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP/output/reduced_gated.csv" }, "gateColumnName": "GFP pos", "quitAfterRun": true, "selectedGate": "TRUE", "numImagesToBeProcessed": 3003 }

schraivo commented 3 years ago

The "Job 0" should actually process 100 images from three tables each (that's what I've entered), but the json says that 3003 images should be processed. Also, the process stops after the first table is done and doesn't go to the second table.

This is the corresponding json file:

{ "tableFiles": [ { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1648_HeLa RIEN NSP1 wt/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1740_HeLa RIEN NSP1 mut1/output/reduced_gated.csv" }, { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1755_HeLa RIEN NSP1 mut2/output/reduced_gated.csv" } ], "minimumFileSizeKiloBytes": 10.0, "maximumFileSizeKiloBytes": 100000.0, "whiteIndexString": "1", "minWhite": 0.0, "maxWhite": 0.17, "greenIndexString": "5", "minGreen": 0.0, "maxGreen": 0.1, "magentaIndexString": "3", "minMagenta": 0.08, "maxMagenta": 1.0, "viewingModality": "Processed Overlay And Individual Channels", "maxNumFiles": 100, "experimentDirectory": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP", "selectedTableFile": { "path": "/g/bdimsort/200807_Dewi_Nsp1/20200830_1634_HeLa RIEN pmaxGFP/output/reduced_gated.csv" }, "gateColumnName": "GFP pos", "quitAfterRun": true, "selectedGate": "TRUE", "numImagesToBeProcessed": 3003 }

Ah now I got it, sorry for the confusion. It indeed saves one file per table. I was confused about the first rows in the json which lists all tables in the table list. But each json will only initiate processing of a single table. Makes sense!

BUT: Still there are 3003 images processed, instead of the number of images I am requesting (in that case I wanted 100 images). There seems to be a bug. It works if there's only one table in the list of tables, but the bug is "active" as soon as there are more than one tables listed.

tischi commented 3 years ago

There seems to be a bug.

Yes, indeed. In fact, I realised that I have to substantially work on a better separation of the user interface (interactive) from the cluster (batch) code. It is a mess... 🙂 I am not sure I will finish it today, but quite certainly by Tuesday. Would that be OK?

schraivo commented 3 years ago

Sure, thanks a lot Tischi!! Enjoy your weekend. I can use the tools in the meanwhile for single tables I guess.

tischi commented 3 years ago

Hi, I put a new version for testing to the usual place: /cba/exchange/imflow/jars/

schraivo commented 3 years ago

OK I'll give it a test tonight or tomorrow!

schraivo commented 3 years ago

@tischi When pressing "select gate" and "preview image", it now doesn't ask for selecting gate and population, instead this is what happens: The select table window pops up, and then when confirming the selected table, there's the following error: ![Uploading Screenshot 2020-10-21 at 15.48.38.png…]()

schraivo commented 3 years ago

Screenshot 2020-10-21 at 15 48 38

tischi commented 3 years ago

You are right. I updated.

  1. Can you please try again?
  2. Are you still running jobs? I cannot update the Fiji on the cluster, probably because you are using it...
tischi commented 3 years ago

ad 2. seems ok now.

schraivo commented 3 years ago

Ah yeah the cluster is currently running. Should I stopp?

Dr. Daniel Schraivogel Postdoctoral Fellow

European Molecular Biology Laboratory (EMBL) Genome Biology Unit Steinmetz lab Meyerhofstrasse 1 D - 69117 Heidelberg

Phone: +49 6221 387 8852 Mobile: +49 170 8641641 daniel.schraivogel@embl.de

On 21. Oct 2020, at 16:37, noreply@github.com wrote:

You are right. I updated.

Can you please try again? Are you still running jobs? I cannot update the Fiji on the cluster, probably because you are using it... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tischi/fccf/issues/18#issuecomment-713623915, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHLRBDQJWN275KLGWGXHVDSL3WYZANCNFSM4SRYU25Q.

tischi commented 3 years ago

nope is ok now. can you test again?

schraivo commented 3 years ago

Yes, gerade dabei Dr. Daniel Schraivogel Postdoctoral Fellow

European Molecular Biology Laboratory (EMBL) Genome Biology Unit Steinmetz lab Meyerhofstrasse 1 D - 69117 Heidelberg

Phone: +49 6221 387 8852 Mobile: +49 170 8641641 daniel.schraivogel@embl.de

On 21. Oct 2020, at 17:37, noreply@github.com wrote:

nope is ok now. can you test again?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tischi/fccf/issues/18#issuecomment-713665801, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHLRBEEOKXYUKEQK26BHTTSL353VANCNFSM4SRYU25Q.

schraivo commented 3 years ago

Unfortunately not yet. This is the error (looks different this time):

Screenshot 2020-10-21 at 17 43 18

tischi commented 3 years ago

Yes, I don't know why but the two other jars that you need where missing from the folder. You also need those (and check that there are no duplicate versions in your jars folder)!

...this will be much easier once we have an official Update site because all of this will be automated with one click...

tischi commented 3 years ago

Just to make sure:

You need those guys

imagej-cluster-0.6.3-SNAPSHPOT.jar 
imagej-utils-0.5.5-SNAPSHOT.jar 
imflow-0.2.0.jar
schraivo commented 3 years ago

Ahh ok. I thought the other jars were gone on purpose :)

On 21. Oct 2020, at 18:02, noreply@github.com wrote:

Yes, I don't know why but the two other jars that you need where missing from the folder. You also need those (and check that there are no duplicate versions in your jars folder)!

...this will be much easier once we have an official Update site because all of this will be automated with one click...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tischi/fccf/issues/18#issuecomment-713682595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHLRBBV5W3JWVSF7OOFQPLSL4AZZANCNFSM4SRYU25Q.

schraivo commented 3 years ago

Wohoo, works :)