Closed bomacchih closed 3 years ago
Hey @ bomacchih,
This error usually occurs as a consequence of something being a bit off with the input files. I'd be happy to give it a look. perhaps you can show me the output of:
head COUNT_FILE.txt
head META_FILE.txt
where you replace X_FILE
with the names of your files. From this I should be able to give at least some advice and hopefully resolve the issue.Best, A
HI Almaan, Thanks for your reply. The attached zip comprises all input files and output files. Tack
Best, Chi-Chih
maize_data.zip https://drive.google.com/file/d/1CKdMrybN0I1Q-2aRMmugY9LVc96KlePc/view?usp=drive_web
On Thu, Jul 15, 2021 at 5:47 PM Alma Andersson @.***> wrote:
Hey @ bomacchih,
This error usually occurs as a consequence of something being a bit off with the input files. I'd be happy to give it a look. perhaps you can show me the output of:
- head COUNT_FILE.txt
- head META_FILE.txt where you replace X_FILE with the names of your files. From this I should be able to give at least some advice and hopefully resolve the issue.
Best, A
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-880555967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU33TQC3BVEMZQU375TTX2VB3ANCNFSM5AMUEG7Q .
Great,
I requested access to the file earlier today, so as soon as that is granted I'll have a look at it.
/A
Thanks for sharing the files, looking at your count file VR_03_revised2.tsv
(which I identified based on your error message above) you have some issues there. The rows in the count file are not the same as in the meta data, more specifically they are numbered from 1 to N (N=number of spots) and then you have a second column named "X" where the spatial barcodes are listed.
I've modified your file and uploaded it here (it's a one-time access link). With the proviso that your other files are correctly formatted, these changes should allow you to execute and run stereoscope
smoothly. I've also included an image below to show you what the old file looked like, and what modifications I've made to it.
Taking some extra time to make sure that your files are correctly formatted is always a good idea, as this is often the culprit of many issues and is usually easily solved.
Good luck with the analysis!
/A
Hi Almaan, Thanks a lot. I did not realize that there is an extra column generated after the table transform. I will pay attention to this issue. Tack
/Chi-Chih
On Fri, Jul 16, 2021 at 2:02 AM Alma Andersson @.***> wrote:
Thanks for sharing the files, looking at your count file VR_03_revised2.tsv (which I identified based on your error message above) you have some issues there. The rows in the count file are not the same as in the meta data, more specifically they are numbered from 1 to N (N=number of spots) and then you have a second column named "X" where the spatial barcodes are listed.
I've modified your file and uploaded it here https://file.io/C9GvoRpSnXE0 (it's a one-time access link). With the proviso that your other files are correctly formatted, these changes should allow you to execute and run stereoscope smoothly. I've also included an image below to show you what the old file looked like, and what modifications I've made to it.
[image: image] https://user-images.githubusercontent.com/8884097/125835149-843a7ab5-c519-4916-bb74-ddea1d38b9d9.png
Taking some extra time to make sure that your files are correctly formatted is always a good idea, as this is often the culprit of many issues and is usually easily solved.
Good luck with the analysis!
/A
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-880902982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU5FRUSCW7NWBITIFIDTX4PBXANCNFSM5AMUEG7Q .
Glad I could help, hope everything goes well!
I'm closing this issue since it seems things are resolved.
/A
Hi Alma, Thanks for your kind help about checking and revising file format. I did "stereoscope run" to generate output files. However, I still got some errors and can not get plots. Please see the following message. The format of mine is not like one on your GitHub. (steroscope) @.** VR_03_revised2_adjusted]$ head Wtsv -n 5 | cut -f 1-5 | column -t 0 1 2 3 TTGAGGCATTTAACTC.1_1 0.056943063 2.6377988e-13 0.17883493 0.32930392 CACCGGAGATATCTCC.1_1 0.007079434 9.747728e-14 0.11178181 0.4537787 CACGTCTATGATGTGG.1_1 0.04331963 2.1303826e-13 4.0616298e-13 0.68091846 TGACCAGCTTCAAAGT.1_1 1.6357763e-12 9.736896e-14 1.7631576e-13 0.32374543
(steroscope) @.** hippo_1]$ stereoscope look -pp
VR_03_revised2_adjusted/Wtsv -o viz -sc i -sb s -nc 7 -c "umap" -g -ms 40
Traceback (most recent call last):
File "/home/macchihlee/.local/bin/stereoscope", line 33, in
I also attached all output files (maize.tar.gz) for your reference. Please help on trouble-shooting.
Thanks in advace.
/Chi-Chih
On Fri, Jul 16, 2021 at 2:02 AM Alma Andersson @.***> wrote:
Thanks for sharing the files, looking at your count file VR_03_revised2.tsv (which I identified based on your error message above) you have some issues there. The rows in the count file are not the same as in the meta data, more specifically they are numbered from 1 to N (N=number of spots) and then you have a second column named "X" where the spatial barcodes are listed.
I've modified your file and uploaded it here https://file.io/C9GvoRpSnXE0 (it's a one-time access link). With the proviso that your other files are correctly formatted, these changes should allow you to execute and run stereoscope smoothly. I've also included an image below to show you what the old file looked like, and what modifications I've made to it.
[image: image] https://user-images.githubusercontent.com/8884097/125835149-843a7ab5-c519-4916-bb74-ddea1d38b9d9.png
Taking some extra time to make sure that your files are correctly formatted is always a good idea, as this is often the culprit of many issues and is usually easily solved.
Good luck with the analysis!
/A
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-880902982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU5FRUSCW7NWBITIFIDTX4PBXANCNFSM5AMUEG7Q .
Hi Alma, I also had a further question about plotting. Here list required files on the protocol
One could also overlay the spots on the tissue image, to see how the spatial patterns relates to the morphology, like we did in our figures. This type of visualization is not a part of the stereoscope package, but we do provide scripts for this.
The material that you need for this would be:
I do not quite know what "Transformation Matrix" file looks like, and how to generate it from 10X spaceranger output (if possible). Could you also kind provide some instruction?
Thanks a lot.
Best,
Chi-Chih
On Fri, Jul 16, 2021 at 2:02 AM Alma Andersson @.***> wrote:
Thanks for sharing the files, looking at your count file VR_03_revised2.tsv (which I identified based on your error message above) you have some issues there. The rows in the count file are not the same as in the meta data, more specifically they are numbered from 1 to N (N=number of spots) and then you have a second column named "X" where the spatial barcodes are listed.
I've modified your file and uploaded it here https://file.io/C9GvoRpSnXE0 (it's a one-time access link). With the proviso that your other files are correctly formatted, these changes should allow you to execute and run stereoscope smoothly. I've also included an image below to show you what the old file looked like, and what modifications I've made to it.
[image: image] https://user-images.githubusercontent.com/8884097/125835149-843a7ab5-c519-4916-bb74-ddea1d38b9d9.png
Taking some extra time to make sure that your files are correctly formatted is always a good idea, as this is often the culprit of many issues and is usually easily solved.
Good luck with the analysis!
/A
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-880902982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU5FRUSCW7NWBITIFIDTX4PBXANCNFSM5AMUEG7Q .
Hi Alma, Thanks for your kind help about checking and revising file format. I did "stereoscope run" to generate output files. However, I still got some errors and can not get plots. Please see the following message. The format of mine is not like one on your GitHub. (steroscope) @. VR_03_revised2_adjusted]$ head Wtsv -n 5 | cut -f 1-5 | column -t 0 1 2 3 TTGAGGCATTTAACTC.1_1 0.056943063 2.6377988e-13 0.17883493 0.32930392 CACCGGAGATATCTCC.1_1 0.007079434 9.747728e-14 0.11178181 0.4537787 CACGTCTATGATGTGG.1_1 0.04331963 2.1303826e-13 4.0616298e-13 0.68091846 TGACCAGCTTCAAAGT.1_1 1.6357763e-12 9.736896e-14 1.7631576e-13 0.32374543 … ------------------------------------------------------------------------------------------------------------- (steroscope) **@.** hippo_1]$ stereoscope look -pp VR_03_revised2_adjusted/Wtsv -o viz -sc i -sb s -nc 7 -c "umap" -g -ms 40 Traceback (most recent call last): File "/home/macchihlee/.local/bin/stereoscope", line 33, in
sys.exit(load_entry_point('stereoscope==0.3.1', 'console_scripts', 'stereoscope')()) File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/main.py", line 16, in main look(args) File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 398, in look crdlist = [get_crd(w,as_he = args.image_orientation) for w in wlist] File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 398, in crdlist = [get_crd(w,as_he = args.image_orientation) for w in wlist] File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 325, in get_crd crd = np.array(crd).astype(float) ValueError: could not convert string to float: 'TTGAGGCATTTAACTC.1_1'
Hello Chi-Chih,
so the issues you are experiencing stems from the fact that your rownames are barcodes and not the expected format [x-coordinate]x[y-coordinate]
which was standard for the old ST1K (the predecessor of Visium) data. When stereoscope
was developed, Visium was not yet a thing. Hence, why I would advice you to replace your rownames with the corresponding spatial coordinates, in the format mentioned above; then the look
module should work fine for you. That's the quick fix, I'm also gonna try to add a command where you can just provide the tissue_positions_list.csv
containing this information to the look
module, but not sure how soon I'll be able to tend to this.
I would also like to make a small disclaimer that the look
module was mainly developed to allow a quick overview of the results, before loading the data into more versatile analysis frameworks. My recommendation is to use something like:
To read in your data and visualize/explore the results
Hope this clarified things, Alma
Hi Alma, I also had a further question about plotting. Here list required files on the protocol One could also overlay the spots on the tissue image, to see how the spatial patterns relates to the morphology, like we did in our figures. This type of visualization is not a part of the stereoscope package, but we do provide scripts for this. The material that you need for this would be: - HE-image - image taken of the tissue, can be scaled but not cropped - Transformation Matrix - matrix that maps spot coordinates to pixel coordinates - Proportion Estimates - output from stereoscope analysis - Mask (optional)- mask to indicate which parts of the area to be included (transparent) resepectively excluded (black) I do not quite know what "Transformation Matrix" file looks like, and how to generate it from 10X spaceranger output (if possible). Could you also kind provide some instruction? Thanks a lot. Best, Chi-Chih
To address this second question, I'm partially gonna reiterate my previous answer; the recipe you list above was given for the older 1K arrays, where a transformation matrix was necessary to map array coordinates to pixel coordinates, the transformation nwas something you got out of a tool that was called the "Spot Detector", which now has been replaced by an automated procedure in spaceranger
. The best option for you would be to use any of the aforementioned software suites above to visualize your results.
You could also visualize these results without any need to use packages, and write the visualization functions yourself. One example of this, where we also visualize stereoscope
results can be found here (`labs/ex4') which contains a computer lab exercise that I compiled for a master-course I've been TA:ing in.
Best, Alma
Hi Alma, Thanks for your detailed reply, and appreciate your input. I will try to load outputs to STutility, since I use it a lot. However, I never load tsv (usually is h5) to STutility, and not quite sure how to combine it with other info (image, json, spot position). Is there a general process to load these files into Seurat format? Any information will be appreciated.
Best, Chi-Chih
On Mon, Jul 19, 2021 at 4:29 PM Alma Andersson @.***> wrote:
Hi Alma, Thanks for your kind help about checking and revising file format. I did "stereoscope run" to generate output files. However, I still got some errors and can not get plots. Please see the following message. The format of mine is not like one on your GitHub. (steroscope) @.*** VR_03_revised2_adjusted]$ head W
*tsv -n 5 | cut -f 1-5 | column -t 0 1 2 3 TTGAGGCATTTAACTC.1_1 0.056943063 2.6377988e-13 0.17883493 0.32930392 CACCGGAGATATCTCC.1_1 0.007079434 9.747728e-14 0.11178181 0.4537787 CACGTCTATGATGTGG.1_1 0.04331963 2.1303826e-13 4.0616298e-13 0.68091846 TGACCAGCTTCAAAGT.11 1.6357763e-12 9.736896e-14 1.7631576e-13 0.32374543 … <#m-7090754811969527823_>
(steroscope) @.** hippo_1]$ stereoscope look -pp VR_03_revised2_adjusted/Wtsv -o viz -sc i -sb s -nc 7 -c "umap" -g -ms 40 Traceback (most recent call last): File "/home/macchihlee/.local/bin/stereoscope", line 33, in sys.exit(load_entry_point('stereoscope==0.3.1', 'console_scripts', 'stereoscope')()) File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/ main.py", line 16, in main look(args) File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 398, in look crdlist = [get_crd(w,as_he = args.image_orientation) for w in wlist] File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 398, in crdlist = [get_crd(w,as_he = args.image_orientation) for w in wlist] File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/look.py", line 325, in get_crd crd = np.array(crd).astype(float) ValueError: could not convert string to float: 'TTGAGGCATTTAACTC.1_1'
Hello Chi-Chih,
so the issues you are experiencing stems from the fact that your rownames are barcodes and not the expected format [x-coordinate]x[y-coordinate] which was standard for the old ST1K (the predecessor of Visium) data. When stereoscope was developed, Visium was not yet a thing. Hence, why I would advice you to replace your rownames with the corresponding spatial coordinates, in the format mentioned above; then the look module should work fine for you. That's the quick fix, I'm also gonna try to add a command where you can just provide the tissue_positions_list.csv containing this information to the look module, but not sure how soon I'll be able to tend to this.
I would also like to make a small disclaimer that the look module was mainly developed to allow a quick overview of the results, before loading the data into more versatile analysis frameworks. My recommendation is to use something like:
- STUtility https://ludvigla.github.io/STUtility_web_site/ : R-based
- Seurat https://satijalab.org/seurat/ : R-based
- scanpy https://scanpy.readthedocs.io/ : Python-based
To read in your data and visualize/explore the results
Hope this clarified things, Alma
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-882354283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU7Y3WVRRCXYEL7C7P3TYPO6LANCNFSM5AMUEG7Q .
Hi Alma, Thanks for your detailed reply, and appreciate your input. I will try to load outputs to STutility, since I use it a lot. However, I never load tsv (usually is h5) to STutility, and not quite sure how to combine it with other info (image, json, spot position). Is there a general process to load these files into Seurat format? Any information will be appreciated.
Best, Chi-Chih
Great that you felt as if your questions were answered. So what I would recommend is to just read in the proportions file as a normal tsv file in R
, that is prop_data <- read.table(PATH_TO_PROPORTIONS,sep="\t",row.names = 1, header = 1)
and then just add this to your object holding the expression data as meta-data, as you normally would (see link).
I've also now added the possibility to provide a command --spatial_folder
to the look
module, where you provide the path to the spatial
folders (meaning the folder that is included in the spaceranger
output, see link), corresponding to your analyzed samples (note they have to be in the same order, if you are only analyzing a single section then just ofc order is not applicable and just provide the single path). In order to be able to use this command, you must pull the latest version and install it.
Best, Alma
Many thanks, Alma, I still have some issues not quite clear.
On Mon, Jul 19, 2021 at 7:03 PM Alma Andersson @.***> wrote:
Hi Alma, Thanks for your detailed reply, and appreciate your input. I will try to load outputs to STutility, since I use it a lot. However, I never load tsv (usually is h5) to STutility, and not quite sure how to combine it with other info (image, json, spot position). Is there a general process to load these files nto Seurat format? Any information will be appreciated.
Best, Chi-Chih
Great that you felt as if your questions were answered. So what I would recommend is to just read in the proportions file as a normal tsv file in R, that is prop_data <- read.table(PATH_TO_PROPORTIONS,sep="\t",row.names = 1, header = 1) and then just add this to your object holding the expression data as meta-data, as you normally would (see link https://ludvigla.github.io/STUtility_web_site/Getting_Started.html).
So far, I do this to add output of W-file from stereoscope.
@.***$SCX6<-prop_data$X6
and use ST.FeaturePlot features. ST.FeaturePlot(VR03, features = "SCX6", cols = c("lightgray", "mistyrose", "red", "dark red", "black"), ncol = 2, pt.size = 2)
I saw an image. So, is there a better way to incorporate w-file data to Seurat object? How can I make sure the order of imported proportion values of each cluster matches the Seurat object?
Is there a way to know what cells are in a given spot interested?
/Chi-Chih
I've also now added the possibility to provide a command --spatial_folder to the look module, where you provide the path to the spatial folders (meaning the folder that is included in the spaceranger output, see link https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/using/count), corresponding to your analyzed samples (note they have to be in the same order, if you are only analyzing a single section then just ofc order is not applicable and just provide the single path). In order to be able to use this command, you must pull the latest version and install it.
Best, Alma
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-882454985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU27XLBXSR4WJL7ZZCDTYQA7LANCNFSM5AMUEG7Q .
Hi,
so I would perhaps like to defer this to STUtility
's gituhub section and/or someone who could help with basic R
problems, as these aren't really questions specifically related to stereoscope
. But you should probably try to use some helper function like AddMetaData
(link) to add the meta data to your object. I would guess that any such helper method automatically checks and arranges the rows in the correct order; but if not I would perhaps do something like:
nm_se <- colnames(se)
nm_pp <-rownames(proportions)
inter <- intersect(nm_se,nm_pp)
se <- SubsetData(se,name = inter)
proportions <- proportions[inter,]
Some useful links are:
Good luck, A
Thanks all of these useful information. Tack and have a nice summer vacation.
/Chi-Chih
On Tue, Jul 20, 2021 at 1:33 PM Alma Andersson @.***> wrote:
Hi,
so I would perhaps like to defer this to STUtility's gituhub section and/or someone who could help with basic R problems, as these aren't really questions specifically related to stereoscope. But you should probably try to use some helper function like AddMetaData (link https://www.rdocumentation.org/packages/Seurat/versions/3.1.4/topics/AddMetaData) to add the meta data to your object. I would guess that any such helper method automatically checks and arranges the rows in the correct order; but if not I would perhaps do something like:
nm_se <- colnames(se)nm_pp <-rownames(proportions)inter <- intersect(nm_se,nm_pp)se <- SubsetData(se,name = inter)proportions <- proportions[inter,]
Some useful links are:
- STUtility github : https://github.com/jbergenstrahle/STUtility
- Biostar discussion on meta data : https://www.biostars.org/p/460935/
Good luck, A
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/almaan/stereoscope/issues/27#issuecomment-883073261, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJJBRU5S3PAX5KJQXDJOXE3TYUDCRANCNFSM5AMUEG7Q .
After running, an error occurred, and W-files were not generated. How do I fix this issue? Tack
[2021-07-15 09:42:11,013 - stsc - INFO ] >> fit st data section(s) : ['/home/macchihlee/ST/spaceranger/steroscope/stereoscope/maize_data/VR_03_revised2.tsv'] Traceback (most recent call last): File "/home/macchihlee/.local/lib/python3.9/site-packages/numpy-1.21.0-py3.9-linux-x86_64.egg/numpy/core/fromnumeric.py", line 57, in _wrapfunc return bound(*args, **kwds) TypeError: '<' not supported between instances of 'int' and 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/macchihlee/.local/bin/stereoscope", line 33, in
sys.exit(load_entry_point('stereoscope==0.3.1', 'console_scripts', 'stereoscope')())
File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/main.py", line 14, in main
run(prs,args)
File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/run.py", line 177, in run
st_data = D.make_st_dataset(args.st_cnt,
File "/home/macchihlee/.local/lib/python3.9/site-packages/stereoscope-0.3.1-py3.9.egg/stsc/datasets.py", line 412, in make_st_dataset
sel = np.argsort(genesize)[::-1]
File "<__array_function__ internals>", line 5, in argsort
File "/home/macchihlee/.local/lib/python3.9/site-packages/numpy-1.21.0-py3.9-linux-x86_64.egg/numpy/core/fromnumeric.py", line 1114, in argsort
return _wrapfunc(a, 'argsort', axis=axis, kind=kind, order=order)
File "/home/macchihlee/.local/lib/python3.9/site-packages/numpy-1.21.0-py3.9-linux-x86_64.egg/numpy/core/fromnumeric.py", line 66, in _wrapfunc
return _wrapit(obj, method, *args, *kwds)
File "/home/macchihlee/.local/lib/python3.9/site-packages/numpy-1.21.0-py3.9-linux-x86_64.egg/numpy/core/fromnumeric.py", line 43, in _wrapit
result = getattr(asarray(obj), method)(args, **kwds)
TypeError: '<' not supported between instances of 'int' and 'str'
(steroscope) [macchihlee@origo maize_data]$ ls
012_cell_cluster.tsv hippo_1 SC_cluster_count_revised.tsv VR_03_revised2.tsv
(steroscope) [macchihlee@origo maize_data]$ cd hippo_1
(steroscope) [macchihlee@origo hippo_1]$ ls
logits.2021-07-11161142.294398.tsv sc_loss.2021-07-11161142.294398.txt stsc.2021-07-11161142.294398.log
R.2021-07-11161142.294398.tsv sc_model.2021-07-11161142.294398.pt
(steroscope) [macchihlee@origo hippo_1]$ ^C
(steroscope) [macchihlee@origo hippo_1]$ ls -la
total 2288
drwxrwxr-x. 1 macchihlee macchihlee 330 Jul 15 09:28 .
drwxrwxr-x. 1 macchihlee macchihlee 146 Jul 11 15:57 ..
-rw-rw-r--. 1 macchihlee macchihlee 133995 Jul 15 09:28 logits.2021-07-11161142.294398.tsv
-rw-rw-r--. 1 macchihlee macchihlee 756305 Jul 15 09:28 R.2021-07-11161142.294398.tsv
-rw-rw-r--. 1 macchihlee macchihlee 1202415 Jul 15 09:28 sc_loss.2021-07-11161142.294398.txt
-rw-rw-r--. 1 macchihlee macchihlee 240999 Jul 15 09:28 sc_model.2021-07-11161142.294398.pt
-rw-rw-r--. 1 macchihlee macchihlee 601 Jul 15 09:28 stsc.2021-07-11161142.294398.log
(steroscope) [macchihlee@origo hippo_1]$ more stsc.2021-07-11161142.294398.log