Open pwalczysko opened 3 years ago
cc @will-moore @sbesson
I am not sure I agree the current behavior is too strict.
As it stands, plate
, well
, image
, dataset
and roi
are reserved column names. The expectation of the API is that if specified the values of these columns must be resolvable into objects found in the hierarchy of the target container.
Supporting the scenario above would require making a decision about whether these columns should be ignored (treated as standard columns) or checked. Looking at a slightly more complex example than the one above, what should be the behavior if a plate was annotated with as CSV as follows?
# header well,plate,s,d,l,d
Well,Plate,Drug,Concentration,Cell_Count,Percent_Mitotic
A1,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,10.1,10,25.4
A2,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,0.1,1000,2.54
A3,INMAC384-DAPI-CM-eGFP_59223_1,DMSO,5.5,550,4
B1,INMAC384-DAPI-CM-eGFP_59223_1,DrugX,12.3,50,44.4
A1,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,10.1,10,25.4
A2,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,0.1,1000,2.54
A3,INMAC384-DAPI-CM-eGFP_59223_2,DMSO,5.5,550,4
B1,INMAC384-DAPI-CM-eGFP_59223_2,DrugX,12.3,50,44.4
In that case, should Plate
be ignored, checked against the
what should be the behavior if a plate was annotated with as CSV as follows?
I would suggest throw an error complaining about non-uniqueness of the well names.
I guess this is already what the same plugin does if I try to pass a CSV on a Dataset with non-unique image names in it ? See below:
# header s,s,d,l,s
Image Name,Dataset Name,ROI_Area,Channel_Index,Channel_Name
Centrin_PCNT_Cep215_20110506_Fri-1608_0_SIR_PRJ.dv,dataset01,0.0469,1,DAPI
Cep215_PCNT_gtub_20110506_Fri-1721_0_SIR_PRJ.dv,dataset01,0.142,2,GFP
Cep215_PCNT_NEDD1_20110505_Thu-1403_0_SIR_PRJ.dv,dataset01,0.093,3,TRITC
Centrin_PCNT_Cep215_20110506_Fri-1608_0_SIR_PRJ.dv,dataset01,0.429,4,Cy5
The confusion here is caused by the apparent discrepancy between the behaviour of this plugin when run on P/D/I as opposed to when run on SPW. The discrepancy starts with the constraint on the SPW case where the well,plate...
columns are compulsory, whereas the corresponding PDI case has no such demands on the CSV with regard to the corresponding image,dataset...
columns, as the example above shows.
Edit: Maybe to try to specify the behaviour in the highlighted case more precisely:
In that case, should Plate be ignored
If the target object is a Plate, then the Plate column should be ignored and the Well column should be insisted on having unique names. If the target object is a Screen, then the Plate column should be taken into account and the Well name uniqueness is not to be insisted upon.
I absolutely agree with @pwalczysko since this error is pretty annoying for me as a user. The "plate" column should be just ignored when populating to a plate and not throw the above mentioned error.
Hello,
I experienced the same issue as @pwalczysko that I cannot process a .csv with a plate
column if I run the script on a plate.
Unfortunately I also cannot process it, as described in the original post, if I run the script with same .csv on a screen containing said plate.
I tried to delete the plate
column in the .csv (also according to the Note panel at the end of the How-To) and run the script again on the plate.
Now I get an error message like this:
Traceback (most recent call last): File "./script", line 192, in
run_script() File "./script", line 184, in run_script message = populate_metadata(client, conn, script_params) File "./script", line 138, in populate_metadata ctx.parse_from_handle_stream(f2) File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1093, in parse_from_handle_stream table = self.create_table() File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_metadata/populate.py", line 1110, in create_table {'omero.group': native_str(group)}) File "/opt/omero/venv_server/lib64/python3.6/site-packages/omero_SharedResources_ice.py", line 393, in newTable return _M_omero.grid.SharedResources._op_newTable.invoke(self, ((repoId, path), _ctx)) omero.InternalException: exception ::omero::InternalException { serverStackTrace = serverExceptionClass = message = null table as argument }
I have no idea where the "null table" exception might come from.
I am using this on a plate
"Overlap 2" under a screen
"Test Screen _ Script".
There are wells L9-24 and M9-24, each containing two images.
Attached is my .csv that I used for testing.
TEST table_02.csv
Am I missing something? Is there a better detailed guide on how to get the metadata into the well-tables (I haven't even started with images yet...) maybe cirvumventing all these subtle pitfalls?
@JensWendt I don't know what's wrong with that CSV. It looks OK. I just did a little test with the first half of your csv (editing the Well names to fit a smaller plate I have for testing) and it worked OK (using omero-metadata 0.9.0
, which looks like what you have - at least populate.py", line 1093
is the same. It shouldn't care about the Plate or the number of Images in a Well. Even when I used Well M24
which doesn't exist on my plate, it didn't fail.
Sorry I can't be more help, but I don't think there's anything wrong with the CSV.
If you have the ability to run on the command line, you can get more verbose logging -vvv
which might help:
$ omero metadata populate Plate:51 --file /path/to/TEST.table_02.csv -vvv
Heyo @will-moore ,
Again, thank you for the incredible quick answer. I found a way to run it from the CLI as you proposed and it gave the following verbose Debug-Log: Script Error.txt
Without knowing the actual structure and code behind it all, my guess from the log would be:
the sript it is able to correctly read the .csv and generate the table with all its rows, but then in the final step when it tries to convert the temporary table it created to an OMERO.table it crashes, for unknown reasons.
For clarification I run the omero-metadata 0.9.0
and OMERO 5.6.1
If you could help out with your background knowledge it would be highly appreciated!
Kind regards
Jens
I wonder if pytables is installed (and working) on the server side? Do you know if populate metadata is working with any data now? Are there existing tables on the server that display OK (as in the screenshot above, when launched from the eye icon in Attachments) or data shown in Tables tab?
@will-moore you were spot on.
Pytables
was installed for the omero-web
venv but not the omero
venv.
I pip-installed the same version into the omero
venv as for the omero-web
and additionally also the pyparser
package, as it sounded like it might be useful as well.
Tried it again via the CLI and it worked.
I will play around with it a bit more now.
Thanks a lot!
When the following command is ueed with a csv as specified below, it ends in error
The same command where the target object is a Screen containing the above Plate is successful.
Suggestion: Make the plugin work in both situations, the present behaviour is too strict.
cc @will-moore @sbesson
The used CSV