Run errors

[abisi]

Hi again,

After #1, I tried executing with an example dataset of mine. I have made a separate copy of the parameters .yml file with, normally, correct raw/processed data paths, with: HDF5 resaving, with fast reader, no fast fusion (Lorenzo was saying that fast fusion was not working yet).

However, I still encounter errors. It could still be a file reading issue. Here's the beginning of the error trace in the Fiji Console:

Also, in the Fiji Log, here's an error that occurred during the same run (but it seems unrelated to the above):

Any ideas on what could be the cause of that?

Thanks! Axel

[lorenzotala]

Is there a *.bfmemo in your example dataset folder? this might cause this issue.

Also, did you map the server on your computer or you gave directly the address of the server?

[lorenzotala]

Can you share the YAML file so that I can see the paths?

Best,

Lorenzo

[abisi]

Hello,

There was no *.bfmemo file, visible or hidden. Yes, I mapped the server.

Here is the yaml file content: parameters_axel_bisi.txt

Edit: I'm currently looking into it and it seems to be a path issue still.

Axel

[lacan]

You say the drive is mapped but it does not poin tto the mapped drive

those line should read something like "Z:\data" analysis_server: "Z:\analysis\Axel_Bisi\\data"

Also why is there an End Of Line between your server and the "data" folder?

[abisi]

Hello,

So, it would seem that the direct address or the mapped address are equivalent (maybe works like aliases?) so this was not an issue I think.

However, I still have errors, which I think relate to the usage of the BIOP dataloader. The issue is that no .xml dataset is defined, therefore no .xml file is found. See attached picture;, where the error shows: java.io.IOException: ZISRAWMETADATA segment expected, found DELETED instead.

I don't think it's a path issue now but more related to the dataloader itself. I tried using BigStitcher directly to define a dataset and the same error as above occured, when using the Zeiss Z.1 dataloader (tested on both Fluorescent Platypus and Bird of Paradise). Here's the error trace I get.

I used to the data using a different dataloader (Zeiss Lightsheet 7) and this creates a dataset without issue. Maybe it is worth going back to that one? I remember seeing posts on image.sc detailing issues with various loaders, at different stages...

Axel

[lacan]

We can start with the older dataloader. It's just the BioFormats one now, as the "Zeiss Lightsheet 7" was a fix that was done in Basel and is not really supported in BigStitcher.

Could I have the dataset that is causing issues placed on Bird of Paradise or Platy?

[abisi]

Hello,

Ah I see, so "Zeiss Lightsheet 7" should not be used anymore? I still have it in my Fiji version in Platy and it works without issue it seems.

I am a bit confused now, as I tried the same loader (Z.1) with different datasets acquired the same way, and creation of the .xml dataset and reasaving is initiated with one, but not the other dataset.

Where would you like me to share the files? They are ~50Gb each.

[lacan]

You could place it in the D: drive in a folder with your name on either Platy or BoP and I will pick it up from there

[abisi]

I copied them in BoP in D:/Axel, it should take about 5 minutes. Weirdly, I cannot load the data with AB077 but it works with AB082. I tried with the Z.1 loader directly, without using the GUI here.

Edit: copied.

[lacan]

So AB077 works for me on a basic BigStitcher install and using the BioFormats Loader And here is part of the log

I am going to test the fastCZI reader now and will let you know

[abisi]

I see, I see the same define views window and log for AB082 (and other files), but for AB077 (and other files) I still get the error of ZISRAWMETADATA segment expected, found DELETED instead when using the Bio-Formats reader.

[lacan]

I still get the error of ZISRAWMETADATA segment expected, found DELETED instead when using the Bio-Formats reader

The Bio-Formats reader uses the FastCZI reader, you need to deactivate it Plugins > Bio-Formats > Bio-Formats Plugins Configuration On the Tab "Formats", scroll all the way down to Zeiss CZI (Quick Start) and uncheck the "enabled" box

[lacan]

Nico has your files and will look into it. As a silver lining, he is very happy to finally encounter this behaviour, as he'd read it in the specs of the format, but never encountered it. Is there something special that weas donw on this file? Like Rewriting an previously failed acquisition with the same name? Something that would explain the difference between AB082 et AB077?

[abisi]

Ok, I will disable it and try with the files that show the issue. I tested with other files, and this error does not occur consistently with the Bio-Formats reader.

With the DELETED in the error, I did think that maybe something went wrong during the acquisition, as you suggest. However, I could not remember now whether it was the case. I tried with other files, some do show the issue, others don't - and in theory they were all acquired the same way.

Edit: I disabled the quick reader and the error persists for the couple files that had it.

[lorenzotala]

Hi Axel, There might be an issue in the output folder generation. Can you verify that the output folders are generated correctly? As I coded it the output will be generated automatically in analysis/username/mouse#. I had similar errors where the folders where not created properly and therefore the xml file could not be saved. Best, Lorenzo

[lacan]

While Nico works on it, I have found a workaround. I opened AB077 in ZEN and just did File > Save As CZI and the new version of the file opens with the fast reader.

[abisi]

@lorenzotala I think there is some confusion on my end. By username, which are you referring to? Is this the user (e.g. Axel, Lorenzo) as an entry of the GUI, or, the username folder of the analysis folder (e.g. Axel_Bisi)? This creates a bit of confusion with the current folder tree structure.

Edit: I did change the raw data path to something custom, because we do not want to test things on the Petersen-Lab/data folder because once things are copied there, it's very annoying to delete. I would like to test the GUI more conveniently, e.g. one unique folder with input and output in it.

For this, where would be the lines in guIBuilder.groovy that I could temporarily hard-code to make sure I look and write things where I think I do?

@lacan Are you referring to the fast reader from the GUI ?

Thank you for your help.

Axel

[lacan]

@lacan Are you referring to the fast reader from the GUI ?

Yes the fast reader works on the CZI I resaved from ZEN. Nico has some trainings this afternoon so I do not know when he can get back to it but I will keep you posted.

[abisi]

Of course, I think I am making progress on my end so I will keep working on it. Would it be possible to be added a collaborator for this repo? That way I could implement modifications, if needed. Thanks!

[lorenzotala]

@lorenzotala I think there is some confusion on my end. By username, which are you referring to? Is this the user (e.g. Axel, Lorenzo) as an entry of the GUI, or, the username folder of the analysis folder (e.g. Axel_Bisi)? This creates a bit of confusion with the current folder tree structure.

Edit: I did change the raw data path to something custom, because we do not want to test things on the Petersen-Lab/data folder because once things are copied there, it's very annoying to delete. I would like to test the GUI more conveniently, e.g. one unique folder with input and output in it.

For this, where would be the lines in guIBuilder.groovy that I could temporarily hard-code to make sure I look and write things where I think I do?

@lacan Are you referring to the fast reader from the GUI ?

Thank you for your help.

Axel

Hi Axel, the user list "Axel, Carl, etc." was just an example. You should use the Username you are currently using in the folder tree of the lab. When I spoke to Lana she told me that in the rcp server you have data stored as "data/AB001/Anatomy/AB001.czi" and that when you do the analysis you will copy things in the following location: "analysis/Username/AB001" where Username should be in your case Axel_Bisi.

If you want to test things just replicate on your rcp server the same folder tree at a specific location and change your yaml accordingly

raw_data_server: : "\\sv-nas1.rcp.epfl.ch\Petersen-Lab\TestGUI\data" analysis_server: "\\sv-nas1.rcp.epfl.ch\Petersen-Lab\TestGUI\analysis" -> just put the base of the analysis folder

The GUI will then create the folder for Axel_Bisi/Mouse# automatically in the analysis_server path "\\sv-nas1.rcp.epfl.ch\Petersen-Lab\TestGUI\analysis\Axel_Bisi\AB001"

This is done at line 617, 788 and 813 on the guiBuilder.groovy

[lorenzotala]

Of course, I think I am making progress on my end so I will keep working on it. Would it be possible to be added a collaborator for this repo? That way I could implement modifications, if needed. Thanks!

I don't think I have the rights to add you but maybe Olivier or Nico do.

Best,

Lorenzo

[abisi]

Ah I see somehow I thought these were different variable names, which was confusing as it's just simple the way you describe it now! Thanks for the clarification.

What you say is correct for the raw "data" folder, however the "analysis" folder is still a bit customized on a user-basis (e.g. Lana and I don't have the same folder organization). Because I don't believe it is possible we can achieve the same folder structure (in analysis), we'll just have to make the output_path flexible, which seems to be the case.

But: I actually managed to make it run after some rearranging. I am currently testing to see if different steps of the processing work okay, but it seems that the overall behaviour works.

If there is anything odd, I'll make another issue. 😄

Last things about the fast CZI reader:

• would it be possible that you share your latest slides (with improvements in speed, etc.). e.g. by email.
• I remember you mentioned that althought it was faster to read, it was incompatible with stitching. Do you remember what you meant by that?

(By the way Olivier already added me as collaborator.)

Thanks again, Axel