Closed will-moore closed 1 year ago
Trying to see how the Rendering Engine uses Pixels.path
and Pixels.name
. Seems like most rendering code is in https://github.com/ome/omero-server/blob/master/src/main/java/ome/services/RenderingBean.java but I don't see any reference to path
there?
It looks like here we're dealing with Pixels object and not accessing the DB directly?
Searching for JDBC
didn't show anything useful.
Also tried search for Duplicate
since I remember that this Database access was used in the duplicate
logic, but couldn't find anything useful.
@jburel - any pointers?
Looking at the Stack Trace at https://github.com/IDR/idr-metadata/issues/659#issuecomment-1567120577 it looks like this might be a good place to start:
at ome.services.blitz.impl.RenderingEngineI.load_async(RenderingEngineI.java:316)
But I don't see blitz
at:
https://github.com/ome/omero-server/tree/master/src/main/java/ome/services
To find the path DuplicateI invokes method in https://github.com/ome/omero-model/blob/master/src/main/java/ome/util/actions/PostgresSqlAction.java#L306 Query is defined in https://github.com/ome/omero-model/blob/master/src/main/resources/ome/util/actions/psql.properties#L5
You need to look into omero-model to understand what is happening
Thanks.
Looks like that model sql.getPixelsNamePathRepo()
is used in getOriginalFilePath()
at
https://github.com/ome/omero-server/blob/master/src/main/java/ome/services/OmeroFilePathResolver.java#L53
but I don't see where that is used in omero-server (e.g. for Duplicate or RenderingEngine).
Possibly relevant to this conversation: while reviewing the histogram server functionality, I was reminded of PixelsService._getPixelBuffer which contains the business logic for selecting the appropriate PixelBuffer (i.e. Bio-Formats vs OMERO Pyramid vs ROMIO) and utilizes the return value of resolver.getOriginalFilePath
Hmmm, OK so that's 4 repos omero-model
, omero-blitz
, omero-server
, omero-romio
that I have to grep to see what uses what. Are there any others?
Searching all for getOriginalFilePath()
finds usage in omero-romio PixelsService.java: makePyramid() getBfReader() getPixelBuffer()
getPixelBuffer()
is used a lot... will look more closely later...
getPixelBuffer()
is used by (ignoring tests):
getBFReader()
is used by:
makePyramid()
is used by:
@will-moore I agree that it is not easy to see what's going on following the split
OK, so lots of places use getOriginalFilePath()
, but can we get that path from the Fileset file list instead of the Pixels table?
We need to be sure that the FIRST OriginalFile of the Fileset is the same as that listed in the Pixels table.
To test this on a sample of images.. I grabbed an Image ID from each study - using the thumbnails from IDR front page:
let ids = [];
$(".viewer_link").map(function(){ids.push(this.href.split("img_detail/")[1].replace("/", ""))});
JSON.stringify(ids.map(i => parseInt(i)))
[14788721,14788631,14271231,14257890,14239635,14249830,14276397,14000892,14239501,14002649,13994856,13981032,13966432,13966192,13965767,13965888,14231132,13461595,13441324,13457674,13966272,13462232,13425213,13444729,13416991,13422206,12922839,13425427,13417058,12689244,12074276,13457423,11576516,12557110,12539666,12529301,13461816,11512177,10560362,12570400,10340801,10647409,12540310,10502961,11407784,9844391,9846131,9846157,9842150,9822152,9846228,9830833,11550334,9837354,10496304,9836841,10501759,9528933,9798438,12814262,12142871,9840230,8833072,13462847,9036345,8343616,9035363,9822058,12922261,6001240,6001463,9753804,5025551,4495402,5514131,4007817,4995115,9846152,4498386,3509480,4007802,13384350,3428080,3489996,3491629,3414085,4996009,1896580,3414398,3262540,3191907,3125701,3137952,3005373,2858229,3261662,3260524,1885643,7948817,1884913,1911858,1851832,1920105,1755287,2044489,1845698,2865040,1811590,2857890,1921258,1313778,106959,96901,353185,930626,719331,16255,223948,1230459]
Use PSQL to get the name
and path
for each of those Images from the Pixels table:
select id, path, name from pixels where image in (14788721,14788631,14271231,14257890,14239635,14249830,14276397,14000892,14239501,14002649,13994856,13981032,13966432,13966192,13965767,13965888,14231132,13461595,13441324,13457674,13966272,13462232,13425213,13444729,13416991,13422206,12922839,13425427,13417058,12689244,12074276,13457423,11576516,12557110,12539666,12529301,13461816,11512177,10560362,12570400,10340801,10647409,12540310,10502961,11407784,9844391,9846131,9846157,9842150,9822152,9846228,9830833,11550334,9837354,10496304,9836841,10501759,9528933,9798438,12814262,12142871,9840230,8833072,13462847,9036345,8343616,9035363,9822058,12922261,6001240,6001463,9753804,5025551,4495402,5514131,4007817,4995115,9846152,4498386,3509480,4007802,13384350,3428080,3489996,3491629,3414085,4996009,1896580,3414398,3262540,3191907,3125701,3137952,3005373,2858229,3261662,3260524,1885643,7948817,1884913,1911858,1851832,1920105,1755287,2044489,1845698,2865040,1811590,2857890,1921258,1313778,106959,96901,353185,930626,719331,16255,223948,1230459);
Then I used a python script test_getOriginalFile_from_fileset.py
based on the query above to get the first original file associated with each Image and printed out any mismatches (after joining path and name). The 1st of each mismatch is from the Fileset.OriginalFile and the 2nd is from the Pixels table:
image_id 1230459
demo_2/2015-11/23/16-25-19.109/Meas_05(2012-07-31_23-29-17)/001001001.flex
demo_2/2015-11/23/16-25-19.109/Meas_01(2012-07-31_18-32-21)/001001001.flex
image_id 1811590
demo_2/2016-05/16/14-53-28.229/uod/idr/filesets/idr0012-fuchs-cellmorph/downloaded/primary/source/www.ebi.ac.uk/huber-srv/cellmorph/source/HT01/HT01F008/HT01F008_A1.tif
demo_2/Blitz-0-Ice.ThreadPool.Server-9/2023-05/11/10-30-57.116/HT01.ome.zarr/OME/METADATA.ome.xml
image_id 1845698
demo_2/2016-05/17/01-31-52.963/uod/idr/filesets/idr0015-UNKNOWN-taraoceans/20150918-tara/RAW_DATA/TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50/slide--S00/chamber--U00--V01/field--X17--Y18/image--L00--S00--U00--V01--J08--E00--O00--X17--Y18--T00--Z00--C00.ome.tif
demo_2/2016-05/17/01-31-52.963/idrdata/idr-metadata/idr0015-UNKNOWN-taraoceans/screenA/patterns/TARA_HCS1_H5_G100001988_G100001989--2013_09_23_19_42_50_chamber--U00--V01.screen
image_id 1896580
demo_2/2016-05/19/00-15-38.492/uod/idr/incoming/idr0016-wawer-bioactivecompoundprofiling/2016-01-19-screens-bbbc022/20585-ERSytoBleed/IXMtest_A01_s1_w3FD54227A-6BE4-460B-8E2C-54B8709F6E33.tif
demo_2/2016-05/19/00-15-38.492/idrdata/idr-metadata/idr0016-wawer-bioactivecompoundprofiling/screens/20585.screen
image_id 1911858
demo_2/2016-05/19/20-47-07.067/Alexis 040413 hMAP screen[2056]/200972432 NS[2668]/2013-04-04T234449Z[7406]/ImageIndex.ColumbusIDX.csv
demo_2/2016-05/19/20-47-07.067/Alexis 040413 hMAP screen[2056]/200972432 NS[2668]/MeasurementIndex.ColumbusIDX.xml
image_id 1921258
demo_2/2016-05/21/00-22-55.102/idrdata/idr-metadata/idr0010-doil-dnadamage/screenA/plates/1-23.pattern
demo_2/Blitz-0-Ice.ThreadPool.Server-3/2023-05/18/20-35-15.516/1-23.ome.zarr/OME/METADATA.ome.xml
image_id 2044489
demo_2/2016-06/15/22-36-23.585/incoming/idr0016-wawer-bioactivecompoundprofiling/2016-02-19-screens-cellimagelibrary/24278-ERSyto/cdp2bioactives_a01_s1_w2edcec6dc-b1e3-4ffc-80da-9b049a89447b.tif
demo_2/2016-06/15/22-36-23.585/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/24278.screen
image_id 2865040
demo_2/2016-09/21/07-41-29.038/filesets/idr0013-neumann-mitocheck/20150916-mitocheck-analysis/analysis/mitocheck/LTValidMitosisSon384Plate03_02--ex2009_03_11--sp2009_01_27--tt17--c6/hdf5/00001_01.ch5
demo_2/2016-09/21/07-41-29.038/metadata/idr0013-neumann-mitocheck/screens/LTValidMitosisSon384Plate03_02--ex2009_03_11--sp2009_01_27--tt17--c6.screen
image_id 3137952
demo_2/2017-03/03/23-39-52.110/20170110-original/200972727[5177]_kinome_YAP_CD44/2014-09-21T053223Z[18861]/ImageIndex.ColumbusIDX.csv
demo_2/2017-03/03/23-39-52.110/20170110-original/200972727[5177]_kinome_YAP_CD44/MeasurementIndex.ColumbusIDX.xml
image_id 3191907
demo_2/2017-03/06/14-24-32.180/uod/idr/filesets/idr0033-rohban-pathways/20170214-original/images/41744/taoe005-u2os-72h-cp-a-au00044859_a01_s1_w1506fb051-c2ad-45db-8a52-674278937a31.tif
demo_2/2017-03/06/14-24-32.180/data/idr-metadata/idr0033-rohban-pathways/screens/41744.screen
image_id 3260524
demo_2/2017-03/13/15-19-51.590/uod/idr/filesets/idr0025-stadler-proteinatlas/20160525-rawdata/10x images plate 1/plate 1 A7B7 ab 7 c10orf58/Sequence/Loop_ 0/A 0_L 0_S 0_U 5_V 0_X 0_Y 0_Job(Seq.Scan) 1 01 01_001_z0_ch00.tif
demo_2/2017-03/13/15-19-51.590/data/idr-metadata/idr0025-stadler-proteinatlas/screens/10x images plate 1.screen
image_id 4007802
demo_2/Blitz-0-Ice.ThreadPool.Server-19/2018-11/26/09-57-43.807/Mmu_E1_CAGTAG1.TM000098_timeFused_blending/SPM00_TM000098_CM00_CM01_CHN00.fusedStack.corrected.shifted.klb
demo_2/Blitz-0-Ice.ThreadPool.Server-19/2018-11/26/09-57-43.807/Mmu_E1_CAGTAG1.TM000036_timeFused_blending/SPM00_TM000036_CM00_CM01_CHN01.fusedStack_yzProjection.corrected.shifted.klb
image_id 4007817
demo_2/Blitz-0-Ice.ThreadPool.Server-24/2018-11/26/10-39-10.551/20181025-ftp/Dataset_3/Unprocessed Image Files/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_t0000_preview.ome.tiff
demo_2/Blitz-0-Ice.ThreadPool.Server-24/2018-11/26/10-39-10.551/patterns/180712_H2B_22ss_Courtney1_20180712-163837_p00_c00_preview.pattern
image_id 4498386
demo_2/Blitz-0-Ice.ThreadPool.Server-9/2019-01/15/17-17-44.268/#2_Analyzed_images/M_nuclei_Exp2_rep2_15min_im2_nuclei3D.tiff
demo_2/Blitz-0-Ice.ThreadPool.Server-9/2019-01/15/17-17-44.268/Exp2_rep2_15min_im2.companion.ome
image_id 4996009
demo_2/Blitz-0-Ice.ThreadPool.Server-1/2019-02/22/16-14-26.593/20140424_HipSci + BJ 3000c_well_OC__2014-04-24T17_01_22-Measurement1/Evaluation2/HipSci_Pipeline_2014-07-31-current.aas
demo_2/Blitz-0-Ice.ThreadPool.Server-1/2019-02/22/16-14-26.593/20140424_HipSci + BJ 3000c_well_OC__2014-04-24T17_01_22-Measurement1/Images/Index.idx.xml
image_id 5025551
demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023/Tonsil 1/Tonsil 1_C00.png
demo_2/Blitz-0-Ice.ThreadPool.Server-6/2023-05/18/09-50-49.857/Tonsil 1.ome.zarr/OME/METADATA.ome.xml
image_id 5514131
demo_2/Blitz-0-Ice.ThreadPool.Server-35/2019-04/07/16-52-18.169/c0011_r0001_ana_early/c0011_r0001_Chr_20170831_Samples20170818_NCAPD3-mEGFP_A_ana-early3_region1_Hoechst.tif
demo_2/Blitz-0-Ice.ThreadPool.Server-35/2019-04/07/16-52-18.169/c0011_r0001_ana_early.companion.ome
image_id 7948817
demo_2/Blitz-0-Ice.ThreadPool.Server-28/2019-07/31/06-41-49.838/Well H02/Transmitted Light - n000000.tif
demo_2/Blitz-0-Ice.ThreadPool.Server-28/2019-07/31/06-41-49.838/Experiment.exp
image_id 9035363
demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-11/25/14-00-58.842/Pos133/4_fam_flour/01.tiff
demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-11/25/14-00-58.842/subpool-6_run-2_EXP-18-BQ3521_round-01.companion.ome
image_id 9822058
demo_2/Blitz-0-Ice.ThreadPool.Server-1/2020-04/23/14-58-14.924/filesets/idr0064-goglia-erkdynamics/20190630-ftp/Plate 3.2/plate_3-2_405_01.nd2.tif_merged.tif
demo_2/Blitz-0-Ice.ThreadPool.Server-1/2020-04/23/14-58-14.924/metadata/idr0064-goglia-erkdynamics/screens/3.2.screen
image_id 10496304
demo_2/Blitz-0-Ice.ThreadPool.Server-6/2020-09/05/15-40-12.840/BR1/TSA2-37/Index.idx.xml
demo_2/Blitz-0-Ice.ThreadPool.Server-6/2020-09/05/15-40-12.840/BR1/TSA2-RT/Index.idx.xml
image_id 12529301
demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-02/05/20-30-14.424/ExperimentA/Plate8/Index.idx.xml
demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-02/05/20-30-14.424/ExperimentA/Plate1/Index.idx.xml
So it looks like there's quite a few where using the first OriginalFile from the Fileset wouldn't give the same path/name as you'd get from querying the Pixels table.
First step (as discussed in IDR meeting today) is to audit the code and check what uses the
path
andname
fields of the Pixels table.