Closed will-moore closed 2 years ago
Hi Will,
sorry, that my response time is so slow at the moment. This is still an important topic for me, it's just the lab work, that is keeping me busy at the moment.
I did a test right now and also really, really like the CSV delimiter sniffing. Thanks @JensWendt for the idea. Worked well for my data, which saves some effort and helps to avoid errors due to the commas in the default image names.
I only encountered one issue, when creating an Excel file with the annotations and then saving it to csv, which is the most likely scenario for our biologists. When I tried to import this csv file, the script struggled with the BOM at the beginning. The info message is green, but the log stucks at the column headers:
script params File_Annotation 1603 IDs [1] Data_Type Plate set ann id 1603 Original File 22636 Copy of plate_test.csv Using delimiter: ; header ['\ufeffPlate', 'Well', 'Image', 'Test1', 'Compound', 'Description']
Maybe this is something you could capture and remove the BOM before processing?
Thank you very much for your effort! Looking forward to enroll the script on our production system once it's finally released.
Anna
Hi Anna,
I also encountered this problem.
My VERY basic solution to this was to add another line after header = data[0]
--> header[0]=str(header[0]).replace("\ufeff","")
From my understanding this is the BOM for UTF-8 encoding.
There are other possible BOMs but I only encountered this one for now.
I am sure there are other more elegant versions out there. But for now this little workaround is helping.
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/multiple-key-value-pair-map-annoation-questions/62976/5
Hi @JensWendt @abhamacher,
I've tried handling the BOM in the same way that we do for Populate_Metadata
script. I've tested this with a sample BOM script I have and it works for that - Hopefully it'll work for yours too?
Hi Will,
I tried it out with two different .csv files and it worked. Seems good to me. Thanks!
See https://forum.image.sc/t/uploading-key-value-pairs-from-csv-files-into-omero-web-for-plates/60202/37 for issues with delimiter sniffing...
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/uploading-key-value-pairs-from-csv-files-into-omero-web-for-plates/60202/58
This pull request has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/uploading-key-value-pairs-from-csv-files-into-omero-web-for-plates/60202/61
Hello @will-moore ,
I checked with two plates in one screen. Both with multiple wells and multiple images per well. There was one combined .csv. I have three remarks:
I had to declare
header=[]
nimg_updated = 0
missing_names = 0
images_by_name = {}
in line 137+ at the beginning of def keyval_from_csv()
, because otherwise the script would complain that I referenced local variables without declaring them first when the return message was created in the end of this method.
The mouseover tooltip for the IDs
script parameter is described as "Plate or Screen ID.". That is misleading,
as I can only select Dataset
or Plate
as data type. I tried to put the Screen ID in that my two plates were under and of course it did not work. We either change it to also allowing Screen
as data type and then getting the objects in the code via listChildren()
- assuming I want to annotate all plates in it - or we have to change the tooltip to accurately represent the wanted input. Maybe something like "ID(s) of your dataset(s) or plate(s)."
The actual out message in the end told me "Added 5 kv pairs to 97/64 files. 396 image names not found." although the script did its job perfectly.
There are multiple things to unpack here. First the number of kv pairs is determined by len(header)-1
this might be better represented by:
num_kv_pairs=len(header)-1
if well_index > -1:
num_kv_pairs-=1
if plate_index > -1:
num_kv_pairs-=1
Second the value after "/". Determined by len(images_by_name)
. Might be more accurate with an image_counter variable comprised by len(images_by_name) + len(wells_by_name)
for each obj iteration plus len(ids)
in the end.
Third, we iterate over the objects (plates) first and then over all the rows. With multiple plates worth of image and well names of course we only match with a fraction of the correct names. Every other name is labeled "not found". This leads to an irritating amount of false negatives with "396 image names not found". To be fair I have no good solution for this from the top of my head. Maybe just skip this error message altogether, but this also is a very sub-par solution.
Well, that concludes my inputs. If I have a super-smart idea how to solve the "images not found" issue I will let you know. If point 1, 2, 3.1 and 3.2 are addressed I feel we are green.
@JensWendt Thanks for that - I've pushed a fix (but not had time to test it yet)... I tried to improve the output message when you select multiple Plates/Datasets. I removed the report on number of columns (since this could be different for each Object) and simply summed up the number of images updated/total.
Worked fine for me. Tested also a file with BOM. Ready to merge fmpov.
This PR contains all commits from https://github.com/mpievolbio-scicomp/omero-scripts/pull/1.
To test: a CSV of this format should behave according to the comment after each row:
A couple of additional points addressed in this PR:
cc @abhamacher Does this look OK? If you could give this a final test and 👍 if looking good, that would be great, thanks.
To test (OME)
Using Plate at https://merge-ci.openmicroscopy.org/web/webclient/?show=plate-16105 (user-3). This has a test
.csv
file attached, using the format above. Sample of it pasted below. Thetarget
column says what should be annotated by each row. Download, inspect and edit the file, then use to run theannotations_scripts/KeyVal_from_csv.py
script. Rows which have animage
name will annotate that Image (Well column is ignored) Rows without animage
will annotate the Well. NB: Can use theRemove_KeyVal.py
script to remove all Well and Image annotations on the Plate if needed.