Trying to make a backend with process.sh, I had issues because my first image had failed QC and my pipeline was set to skip the rest if it did so, so there was only a small subset of the usual amount of data present in that folder- only the Image.csv (with a smaller number of columns), no object CSVs. It therefore for all images only added the common columns, which led to reasonable errors when it went to aggregate the object tables at the end and could not find any.
I hacked around it by deleting all the folders before the first well that actually had been plated with cells, but it seems to me that
1) in creation of the backend we want to pull in all the data that's there, we can deal with NaNs from missing data later
2) there's a reasonable chance of this happening again, as well A01 (and other edge/corner wells) are often not plated into in smaller experiments but in a large fraction of those cases the whole plate is just imaged anyway.
Feel free to disagree though, that's why I phrased it as a question.
End of the error string below, I doubt it's helpful but just in case
(builtins.OSError) /home/ubuntu/efs/{redacted}/workspace/software/cytominer_scripts/.4b21aa7e-6e45-11e7-8ea1-0e60212e428a:1: expected 218 columns but found 568 - extras ignored
[SQL: 'sqlite3 -nullvalue \'\' -separator , -cmd .import "/home/ubuntu/efs/{redacted}/workspace/software/cytominer_scripts/.4b21aa7e-6e45-11e7-8ea1-0e60212e428a" "Image" /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite']
[Fri Jul 21 16:41:41 UTC 2017] Looking up AU00027623.sqlite on permanent store
[Fri Jul 21 16:41:41 UTC 2017] /home/ubuntu/bucket/projects/{redacted}/workspace/backend/2017_07_12_Batch1/AU00027623/AU00027623.sqlite not found
[Fri Jul 21 16:41:41 UTC 2017] Creating /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite
real 127m40.736s
user 62m50.242s
sys 4m23.562s
[Fri Jul 21 18:49:21 UTC 2017] Indexing /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite
Error: near line 3: no such table: main.Cells
Error: near line 4: no such table: main.Cytoplasm
Error: near line 5: no such table: main.Nuclei
real 0m0.054s
user 0m0.009s
sys 0m0.023s
[Fri Jul 21 18:49:22 UTC 2017] Aggregating /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.sqlite
Error in rsqlite_send_query(conn@ptr, statement) : no such table: cells
Calls: %>% ... initialize -> initialize -> rsqlite_send_query -> .Call
Execution halted
real 0m0.993s
user 0m0.603s
sys 0m0.056s
[Fri Jul 21 18:49:23 UTC 2017] /home/ubuntu/ebs_tmp/2017_07_12_Batch1/AU00027623//AU00027623.csv not created / does not exist. Exiting.
shntnu
commented
7 years ago
Trying to make a backend with
process.sh, I had issues because my first image had failed QC and my pipeline was set to skip the rest if it did so, so there was only a small subset of the usual amount of data present in that folder- only the Image.csv (with a smaller number of columns), no object CSVs. It therefore for all images only added the common columns, which led to reasonable errors when it went to aggregate the object tables at the end and could not find any.I hacked around it by deleting all the folders before the first well that actually had been plated with cells, but it seems to me that 1) in creation of the backend we want to pull in all the data that's there, we can deal with NaNs from missing data later 2) there's a reasonable chance of this happening again, as well A01 (and other edge/corner wells) are often not plated into in smaller experiments but in a large fraction of those cases the whole plate is just imaged anyway.
Feel free to disagree though, that's why I phrased it as a question.
End of the error string below, I doubt it's helpful but just in case