KeyError in reader2.py

[adescoeudres]

Hi, I am trying to run the HMRF on some data for which I have the raw expression matrix and the spatial locations as .csv files. I am basically running the following:

raw_matrix <- data.table::fread(fs::path(data_path, 'matrix.csv'), header=FALSE)
spatial_locations <- data.table::fread(fs::path(data_path, 'coords.csv'))

sample_gobject <- createGiottoObject(raw_exprs = Matrix::as.matrix(data.table::transpose(raw_matrix), rownames.value=as.character(1:dim(raw_matrix)[2])), spatial_locs = spatial_locations, instructions = myinst)

# filter genes and cells, normalize, add gene & cell statistics
# ...
# Dimensional reduction (HVG calculation, PCA, UMAP, tSNE, create spatial network)
# ...

# silhouette score
spatial_genes = silhouetteRank(gobject = sample_gobject)
my_spatial_genes <- spatial_genes[1:100,]$gene

HMRF_spatial_genes=doHMRF(gobject=sample_gobject,
                          expression_values='scaled', 
                          spatial_genes=my_spatial_genes, 
                          k=n_cluster,   
                          spatial_network_name="spatial_network", 
                          betas=c(0, 20, 4),  
                          output_folder=paste0(dir.output, '/', 'Spatial_genes/SG_topgenes_k_scaled'))

However, from doHMRF(), I only get the following output and error:

 expression_matrix.txt already exists at this location, will be overwritten                                                                                                                        

 spatial_genes.txt already exists at this location, will be overwritten                                                                                                                            

 spatial_network.txt already exists at this location, will be overwritten 

 spatial_cell_locations.txt already exists at this location, will be overwritten 
[1] "/home/alice/miniconda3/envs/Giotto-env/bin/python /home/alice/miniconda3/envs/Giotto-env/lib/R/library/Giotto/python/reader2.py -l \"/gcm-lfs1/alice/workspace/projSDB/results/maynard/151508/giotto/Spatial_genes/SG_topgenes_k_scaled/spatial_cell_locations.txt\" -g \"/gcm-lfs1/alice/workspace/projSDB/results/maynard/151508/giotto/Spatial_genes/SG_topgenes_k_scaled/spatial_genes.txt\" -n \"/gcm-lfs1/alice/workspace/projSDB/results/maynard/151508/giotto/Spatial_genes/SG_topgenes_k_scaled/spatial_network.txt\" -e \"/gcm-lfs1/alice/workspace/projSDB/results/maynard/151508/giotto/Spatial_genes/SG_topgenes_k_scaled/expression_matrix.txt\" -o \"/gcm-lfs1/alice/workspace/projSDB/results/maynard/151508/giotto/Spatial_genes/SG_topgenes_k_scaled/result.spatial.zscore\" -a test -k 7 -b 0 20 4 -t 1e-10 -z none -s 100 -i 100"
Done reading expression
Done reading location
Done reading graph
Start calculating independent regions
Edges for component 0
Reading Graph...
Computing Clique...
Maximum Clique Size Found: 3
Vertices in the Clique:
1 0 432 

231 milliseconds (excluding I/O).
7 coloring found using DSatur.
Applying Iterated Greedy Improvement...
Found Better Coloring - 6
Found Better Coloring - 5
Applying Local Search...
Final Coloring of graph possible with 5 colors.
Colors of Vertices: 
Traceback (most recent call last):
  File "/home/alice/miniconda3/envs/Giotto-env/lib/R/library/Giotto/python/reader2.py", line 230, in <module>
    new_blocks.append(blocks[b])
KeyError: 2535

What could be the issue here? I checked the blocks dictionary, and really only this key 2535 seems to be missing.

Thanks in advance for your help!

[RubD]

@gcyuan or @pacificma any suggestions?

[pacificma]

Hi @adescoeudres ,

Thank you for reporting this issue! We are updating our HMRF function, and will not use python functions in the next version including the 'reader2.py' which caused issues in your example. The core algorithm of the new HMRF function will be the same as the previous one, but the layout will be somehow different. Currently we have named the new version as 'initHMRF_V2', and 'doHMRF_V2' in Giotto Master branch, and we plan to replace the current function with the new functions in the next update of Giotto version. For instruction of the new functions please check the example here https://bitbucket.org/qzhudfci/smfishhmrf-r/src/master/TRANSITION.md

Please let us know if you have any further questions.

[adescoeudres]

hi @pacificma,

Thank you for looking into this. I have adapted my code to use the new functions, and this seems to have solved also some different problems I was experiencing.

However, with the function doHMRF_V2, I am now getting a new error in some cases, namely: HMRF is stopping at large beta >= 0, numerical error occurs, results of smaller betas were stored Error in class(result.hmrf) <- append(class(result.hmrf), "HMRFoutput") : attempt to set an attribute on NULL This is occurring regardless of the size of the betas (the function call in this case is res = doHMRF_V2(hmrf, betas = c(0,1,2))). Is this a quick fix, or should I open a new issue?

Interestingly, the error only occurs in some of the samples for which I am trying to run Giotto.

Thanks in advance for your help.