higlass cannot find UUID after higlass-ingest

[cafletezbrant]

I have a collection of cooler files and run

docker exec higlass-container python higlass-server/manage.py ingest_tileset --filename /tmp/{$COOLER} --filetype cooler

Where $COOLER is one of my files. I then ping the server to check if ingestion worked and get results for each of my files. Example file results:

{"uuid":"eJzf9xCxR_Kq2hQgQrmAxA","datafile":"http://localhost:8888/api/v1/tilesets/media/uploads/GSM3576787_iPSCORE_2_1_iPSC_GSM3576787_iPSCORE_2_1.iPSC.hic_chr22_5kb.txt_S0700_glICi95.cool","filetype":"cooler","datatype":"matrix","name":"GSM3576787_iPSCORE_2_1_iPSC_GSM3576787_iPSCORE_2_1.iPSC.hic_chr22_5kb.txt_S07001_A_F_18_0.bg2.cool","coordSystem":"","coordSystem2":"","created":"2020-11-20T22:13:45.467503Z","project":"XV0x23ACSquK8DAPcUIKVQ","project_name":"","description":"","private":false}

Being suggested that we then check if we can pull data, I get this surprising result

> curl http://localhost:8888/api/v1/tileset_info/?d="'eJzf9xCxR_Kq2hQgQrmAxA'" {"'eJzf9xCxR_Kq2hQgQrmAxA'": {"error": "No such tileset with uid: 'eJzf9xCxR_Kq2hQgQrmAxA'"}}

Weirdly, I can see the file from the add-track menu, but it never loads any data and just hangs. What am I doing wrong?

[pkerpedjiev]

Hey, a couple thoughts:

1. It looks like you're including quotes in the curl command, maybe try:

curl http://localhost:8888/api/v1/tileset_info/?d=eJzf9xCxR_Kq2hQgQrmAxA

2. Is the cooler file multires? The extension makes me think that maybe it's not? If it's not, you can make it multires by using:

cooler zoomify $COOLER

3. If those don't solve your issue, could you take a screenshot of the browser console when loading the file? Also, can you also paste the last few lines of output from your higlass server log file (docker exec higlass-container cat /data/log/hgs.log)?

[cafletezbrant]

OK, this is weirder. Thanks for this, switching to multiresolution fixes things such that curl can find data:

"uuid":"b4lnBVzeSGuKlhdr7AHZwQ","datafile":"http://localhost:8888/api/v1/tilesets/media/uploads/GSM3576787_iPSCORE_2_1_iPSC_GSM3576787_iPSCORE_2_1.iPSC.hic_chr22_5kb.txt_S070_AGdKfsc.mcool","filetype":"cooler","datatype":null,"name":"GSM3576787_iPSCORE_2_1_iPSC_GSM3576787_iPSCORE_2_1.iPSC.hic_chr22_5kb.txt_S07001_A_F_18_0.bg2.mcool","coordSystem":"","coordSystem2":"","created":"2020-11-28T17:55:01.218756Z","project":"LwaP7pegSQ-I5aonvqm2Pw","project_name":"bnbc","description":"","private":false}

Now I can do

curl "http://localhost:8888/api/v1/tileset_info/?d=b4lnBVzeSGuKlhdr7AHZwQ" {"b4lnBVzeSGuKlhdr7AHZwQ": {"resolutions": [5000, 10000, 20000, 40000, 80000, 160000], "transforms": [], "max_pos": [51304566, 51304566], "min_pos": [1, 1], "chromsizes": [["chr22", 51304566]], "name": "GSM3576787_iPSCORE_2_1_iPSC_GSM3576787_iPSCORE_2_1.iPSC.hic_chr22_5kb.txt_S07001_A_F_18_0.bg2.mcool", "datatype": null, "coordSystem": "", "coordSystem2": ""}}

But now I cannot find the *mcool files in the add-track menu for the UI (I deleted via superuser all the old files I uploaded, so only the mcool files are here, and they are visible from superuser as well). Thanks!

[cafletezbrant]

Sorry, in reply to your other comment I am not using the browser to load file, but the Docker-level instructions on the front page of this repo. Here is the command I am using to load

docker exec higlass-container python higlass-server/manage.py ingest_tileset --filename /tmp/{$COOLER} --filetype cooler

[cafletezbrant]

Finally, log messages (I am in US California, I assume these timestamps are Docker-specific):

[28/Nov/2020 16:54:52] ERROR [django.request:228] Internal Server Error: /api/v1/tileset_info/ Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 126, in _get_response response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python3.6/dist-packages/django/core/handlers/base.py", line 124, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/usr/local/lib/python3.6/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view return view_func(*args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/django/views/generic/base.py", line 68, in view return self.dispatch(request, args, kwargs) File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 495, in dispatch response = self.handle_exception(exc) File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 455, in handle_exception self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.6/dist-packages/rest_framework/views.py", line 492, in dispatch response = handler(request, *args, *kwargs) File "/usr/local/lib/python3.6/dist-packages/rest_framework/decorators.py", line 54, in handler return func(args, **kwargs) File "./tilesets/views.py", line 623, in tileset_info tileset_object.datafile.path File "/usr/local/lib/python3.6/dist-packages/clodius/tiles/cooler.py", line 556, in tileset_info (f, info) = make_mats(filepath) File "/usr/local/lib/python3.6/dist-packages/clodius/tiles/cooler.py", line 519, in make_mats info = _get_info_multi_v1(filepath) File "/usr/local/lib/python3.6/dist-packages/clodius/tiles/cooler.py", line 179, in _get_info_multi_v1 raise ValueError("The max_zoom attribute is missing.") ValueError: The max_zoom attribute is missing. [28/Nov/2020 17:51:30] WARNING [django.security.SuspiciousSession:115] Session data corrupted [28/Nov/2020 17:51:32] WARNING [django.security.SuspiciousSession:115] Session data corrupted [28/Nov/2020 17:51:33] WARNING [django.security.SuspiciousSession:115] Session data corrupted [28/Nov/2020 18:31:19] WARNING [django.security.SuspiciousSession:115] Session data corrupted [28/Nov/2020 18:31:19] WARNING [django.security.SuspiciousSession:115] Session data corrupted [28/Nov/2020 18:31:23] WARNING [django.security.SuspiciousSession:115] Session data corrupted

[pkerpedjiev]

Oh, that's easy. You just have to set datatype of your dataset to matrix. You should be able to change that directly in the superuser interface.

I presume the log messages are from the pre-multires version?

[cafletezbrant]

Setting datatype worked, thank you. This software is quite nice!

Yes, those logs are pre-multires version.

Bc you have resolved this issue of accessing data after ingestion, I am going to close this issue. Thanks much!

[pkerpedjiev]

Great, glad to hear that helped :-)

Just FYI, there's also the higlass-manage script that abstracts away the docker commands to simplify starting an instance and ingesting data. Might be useful to you.