Open will-moore opened 8 months ago
Tested this (originally posted at https://github.com/IDR/idr-metadata/issues/675#issuecomment-1881406444)...
For idr0009 - need to add scanr.skip_missing_wells=false
to a .bfoptions
file for every Plate (964 plates)!
E.g. ?show=well-607352 plate has missing Wells. Trying to view any image after missing Wells shows wrong Image or error.
Fileset template prefixes - test with just a couple from that Plate and the next
http://localhost:1080/webclient/?show=plate-2703 http://localhost:1080/webclient/?show=plate-2705
bash-4.2$ cat idr0009_filesetprefix.csv
demo_2/2015-11/26/07-41-09.642
demo_2/2015-11/26/07-41-14.425
For each, create experiment_descriptor.xml.bfoptions
for f in $(cat idr0009_filesetprefix.csv); do echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$f/experiment_descriptor.xml.bfoptions"; done
This has created .bfoptions file... for http://localhost:1080/webclient/?show=plate-2705
bash-4.2$ ls -alh /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 112K
drwxrwxr-x. 3 omero-server omero-server 153 Jan 8 15:43 .
drwxrwxr-x. 72 omero-server omero-server 8.0K Nov 26 2015 ..
lrwxrwxrwx. 1 omero-server omero-server 97 Nov 26 2015 AcquisitionLog.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/AcquisitionLog.dat
drwxrwxr-x. 2 omero-server omero-server 72K Nov 26 2015 data
lrwxrwxrwx. 1 omero-server omero-server 104 Nov 26 2015 experiment_descriptor.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.dat
lrwxrwxrwx. 1 omero-server omero-server 104 Nov 26 2015 experiment_descriptor.xml -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-10--2007-04-10/experiment_descriptor.xml
-rw-r--r--. 1 omero-server omero-server 31 Jan 8 15:43 experiment_descriptor.xml.bfoptions
The bfmemo exists for some Plates, e.g. first Plate of idr0009 on idr-testing:
ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-22-16.896
total 448K
drwxrwxr-x. 2 omero-server omero-server 47 Nov 30 17:36 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan 8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 448K Nov 30 17:36 .experiment_descriptor.xml.bfmemo
and for Plate above http://localhost:1080/webclient/?show=plate-2705 with some missing Wells
bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server 47 Jan 8 15:37 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan 8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan 8 15:37 .experiment_descriptor.xml.bfmemo
but not for other plates with missing Wells - e.g. first plate above
bash-4.2$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642
total 0
drwxrwxr-x. 2 omero-server omero-server 6 Jan 8 15:58 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan 8 15:58 ..
lets try to delete .bfmemo and see if it fixes problem...
bash-4.2$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/.experiment_descriptor.xml.bfmemo
Now, we can no-longer view ANY images on http://localhost:1080/webclient/?show=plate-2705
serverExceptionClass = ome.conditions.ResourceError
message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml
👎
After I delete the bfoptions file created above
$ rm /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425/experiment_descriptor.xml.bfoptions
This allows images to be viewable again and a new .bfmemo is created
$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-14.425
total 444K
drwxrwxr-x. 2 omero-server omero-server 47 Jan 8 16:41 .
drwxrwxr-x. 5 omero-server omero-server 66 Jan 8 15:58 ..
-rw-rw-r--. 1 omero-server omero-server 443K Jan 8 16:41 .experiment_descriptor.xml.bfmemo
To get more stack trace, repeat..
As omero-server on idr-testing:omeroreadwrite
http://localhost:1080/webclient/?show=plate-2703 is currently viewable, except for last Wells
Write bfoptions... and check...
bash-4.2$ echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/experiment_descriptor.xml.bfoptions"
bash-4.2$ ls -alh /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/
total 108K
drwxrwxr-x. 3 omero-server omero-server 153 Jan 10 11:11 .
drwxrwxr-x. 72 omero-server omero-server 8.0K Nov 26 2015 ..
lrwxrwxrwx. 1 omero-server omero-server 97 Nov 26 2015 AcquisitionLog.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-09--2007-04-08/AcquisitionLog.dat
drwxrwxr-x. 2 omero-server omero-server 68K Nov 26 2015 data
lrwxrwxrwx. 1 omero-server omero-server 104 Nov 26 2015 experiment_descriptor.dat -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-09--2007-04-08/experiment_descriptor.dat
lrwxrwxrwx. 1 omero-server omero-server 104 Nov 26 2015 experiment_descriptor.xml -> /idr/filesets/idr0009-simpson-secretion/20150507-VSVG/VSVG/0305-09--2007-04-08/experiment_descriptor.xml
-rw-r--r--. 1 omero-server omero-server 31 Jan 10 11:11 experiment_descriptor.xml.bfoptions
Delete bfmemo
rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/.experiment_descriptor.xml.bfmemo
Try to view any image from Plate in webclient... E.g. http://localhost:1080/webclient/?show=image-1315721
2024-01-10 11:11:44,870 INFO [ omero.cmd.SessionI] (l.Server-7) Removed servant from adapter: 9505cb38-edc4-4c3e-8564-453be22fd71bomero.api.IQuery
2024-01-10 11:11:46,103 WARN [ o.s.b.repo.ManagedReaderSecurityCheck] (.Server-24) reader for /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/experiment_descriptor.xml uses file from repository cdf35825-def1-4580-8d0b-9c349b8f78d6 that is not readable from database
2024-01-10 11:11:46,104 DEBUG [ loci.formats.Memoizer] (.Server-24) start[1704885104459] time[1644] tag[loci.formats.Memoizer.setId]
2024-01-10 11:11:46,104 ERROR [ ome.io.bioformats.BfPixelBuffer] (.Server-24) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/experiment_descriptor.xml
2024-01-10 11:11:46,105 ERROR [ ome.io.nio.PixelsService] (.Server-24) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/experiment_descriptor.xml
java.lang.RuntimeException: ome.conditions.SecurityViolation: reader for /data/OMERO/ManagedRepository/demo_2/2015-11/26/07-41-09.642/experiment_descriptor.xml uses file from repository cdf35825-def1-4580-8d0b-9c349b8f78d6 that is not readable from database
at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
at jdk.internal.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
This is failing because the .bfoptions
file is not registered in the DB.
Attempting to create a .bfoptions
OriginalFile via the OMERO API only allows it to be created in the /OMERO/Files/
location and not alongside the data in /OMERO/ManagedRepository/
.
Try to use SQL instead...
addbfoptions.sql
:
CREATE OR REPLACE FUNCTION add_origfile_to_fileset(
fileset_id bigint,
name character varying,
uuid character varying)
RETURNS integer AS
$BODY$
DECLARE
new_event integer;
old_owner integer;
old_group integer;
old_perms integer;
new_file integer;
files_count integer;
prefix character varying;
BEGIN
select _current_or_new_event() into new_event;
select
owner_id, group_id, permissions, templateprefix
into
old_owner, old_group, old_perms, prefix
from fileset where id = fileset_id;
select count(id) from filesetentry where fileset = fileset_id into files_count;
insert into originalfile (id, permissions, creation_id, group_id, owner_id, update_id, mimetype, repo, path, name)
values (nextval('seq_originalfile'), old_perms, new_event, old_group, old_owner, new_event,
'application/octet-stream', 'cdf35825-def1-4580-8d0b-9c349b8f78d6', prefix, uuid || name)
returning id into new_file;
insert into filesetentry
(id, permissions, creation_id, group_id, owner_id, update_id, fileset, originalfile, fileset_index, clientpath)
values (nextval('seq_filesetentry'), old_perms, new_event, old_group, old_owner, new_event,
fileset_id, new_file, files_count, name);
RETURN new_file;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
$ psql -U omero -d idr -h $DBHOST -f addbfoptions.sql
CREATE FUNCTION
Then:
$ psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset(19312, 'experiment_descriptor.xml.bfoptions', '9630ba1e-ed3a-42e3-9296-59ccf23a7039')"
add_origfile_to_fileset
-------------------------
63726151
(1 row)
on idr0125-pilot, as omero-server...
$ cat idr0009_filesets.csv
16604,demo_2/2015-11/26/07-41-14.425/
16605,demo_2/2015-11/26/07-41-25.452/
16603,demo_2/2015-11/26/07-41-09.642/
...
$ cat idr0009_filesets.csv | wc
1035 1034 39126
We want to:
export $SECRET=9630ba1e-ed3a-42e3-9296-59ccf23a7039
for r in $(cat idr0009_filesets.csv); do
fsid=$(echo $r | cut -d',' -f1)
prefix=$(echo $r | cut -d',' -f2 | sed 's/.$//')
echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$prefix/experiment_descriptor.xml.bfoptions";
psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'experiment_descriptor.xml.bfoptions', '$SECRET')"
rm -f /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/.experiment_descriptor.xml.bfmemo
done
This issue has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/progressively-attach-timeframes-to-images-in-omero/93506/4
Final test on pilot-idrngff
...
wc idr0009_filesets.csv
1034 1034 39125 idr0009_filesets.csv
vi addbfoptions.sql # see above
$ psql -U omero -d idr -h $DBHOST -f addbfoptions.sql
CREATE FUNCTION
$ export SECRET=41fe4790-5848-4ae9-bc08-c84157af8191
for r in $(cat idr0009_filesets.csv); do
fsid=$(echo $r | cut -d',' -f1)
prefix=$(echo $r | cut -d',' -f2 | sed 's/.$//')
echo "scanr.skip_missing_wells=false" > "/data/OMERO/ManagedRepository/$prefix/experiment_descriptor.xml.bfoptions";
psql -U omero -d idr -h $DBHOST -c "select add_origfile_to_fileset($fsid, 'experiment_descriptor.xml.bfoptions', '$SECRET')"
rm -f /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/$prefix/.experiment_descriptor.xml.bfmemo
done
All images viewable once memo files regenerated (quite quick in webclient)
Josh "SQL looks reasonable enough. I'd say we should really move this to a server method long-term though. (I'd probably have done max()+1
rather than select count(id) from filesetentry where fileset = fileset_id into files_count;
but that's likely not too important assuming there aren't any usedFiles list with nulls
)
Currently on idr-next (prod120) idr0009 Plates are not handling missing Wells correctly.
Seb: “The problem is that this data was originally loaded with a version of the ScanR reader that represents missing wells as black images while the default behavior in Bio-Formats is to represent these as sparse plates. When this was backported to OME Bio-Formats, an option was introduced to support both behaviors - see https://bio-formats.readthedocs.io/en/latest/formats/options.html I think the action is to update all plates in this study to set scanr.skip_missing_wells to false , delete & regenerate the memo files and retest”