Keep cache synced with files

[slinnarsson]

When a loom file is updated, this is not currently reflected in the cache.

[JobLeonard]

We already keep track of when a file was last modified, since we want to be able to sort/filter by it in the viewer. However, we don't cache the result of that. So we first need to add that (which will also save us needless calls to make_list_entry). Then we can add a simple comparison check before deciding to serve the cached loom connection, at this point in the code.

I'll have a go at implementing it, should be done quickly.

• [x] cache make_list_entry results
• [x] for cached looms, compare last_modified of file to last_modified in cache, refresh cache if mismatch

[slinnarsson]

Also, when a file is uploaded through the API, the cache should be refreshed, so that it is ready when requested. This will save latency on first request esp for large datasets.

-- Sten Linnarsson, PhD Professor of Molecular Systems Biology Karolinska Institutet Unit of Molecular Neurobiology Department of Medical Biochemistry and Biophysics Scheeles väg 1, 171 77 Stockholm, Sweden<x-apple-data-detectors://1/0> +46 8 52 48 75 77<tel:+46%208%2052%2048%2075%2077> (office) +46 70 399 32 06<tel:+46%2070%20399%2032%2006> (mobile)

8 dec. 2016 kl. 07:13 skrev Job van der Zwan notifications@github.com<mailto:notifications@github.com>:

We already keep track of when a file was last modifiedhttps://github.com/linnarsson-lab/Loom/blob/d592775e6f74715b7d6180d1b8549b826179445f/python/loompy/loom_cache.py#%5BL121-L128%5D, since we want to be able to sort/filter by it in the viewer. However, we don't cache the result of that. So we first need to add that (which will also save us needless calls to make_list_entryhttps://github.com/linnarsson-lab/Loom/blob/d592775e6f74715b7d6180d1b8549b826179445f/python/loompy/loom_cache.py#%5BL96-L120%5D). Then we can add a simple comparison check before deciding to serve the cached loom connection, at this point in the codehttps://github.com/linnarsson-lab/Loom/blob/d592775e6f74715b7d6180d1b8549b826179445f/python/loompy/loom_cache.py#%5BL150-L151%5D.

I'll have a go at implementing it, should be done quickly.

• cache make_list_entry results
• for cached looms, compare last_modified of file to last_modified in cache, refresh cache if mismatch

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/77#issuecomment-265661716, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKagwNc01fyJ80QNwBqix7YZHLzsl8Kks5rF5_zgaJpZM4LGop8.

[JobLeonard]

I think I've implemented both now, but maybe you can test the upload one to make sure

[JobLeonard]

Is it working as expected, Sten? I'll implement a few more small improvements before pushing anything to the server anyway, but it's good policy to try to not break anything on a Friday afternoon :P

[slinnarsson]

I uploaded a lot of file (40+) yesterday, so probably yes!

/Sten

-- Sten Linnarsson, PhD Professor of Molecular Systems Biology Karolinska Institutet Unit of Molecular Neurobiology Department of Medical Biochemistry and Biophysics Scheeles väg 1, 171 77 Stockholm, Sweden +46 8 52 48 75 77 (office) +46 70 399 32 06 (mobile)

On 9 Dec 2016, at 16:28, Job van der Zwan notifications@github.com<mailto:notifications@github.com> wrote:

Is it working as expected, Sten? I'll implement a few more small improvements before pushing anything to the server anyway, but it's good policy to try to not break anything on a Friday afternoon :P

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/77#issuecomment-266040848, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKag3DKmUiYtXkmP5PB8k9klDFKplVkks5rGXOBgaJpZM4LGop8.

[JobLeonard]

Locally, after git pulling my updates? To clarify: I only pushed the code changes to github, not to the server, because I wanted to make sure nothing was broken.

[slinnarsson]

Aha, no to the server. I can try locally on Monday.

Note I removed the type information (schema) from loompy and bravely/stupidly pushed it to git and pip. I did test on my own machine but probably there are bugs left. The new version also fixes (again) the bug that makes some attributes look like b'Actb' etc. I'll keep fixing Unicode bugs until I retire.

Sten

-- Sten Linnarsson, PhD Professor of Molecular Systems Biology Karolinska Institutet Unit of Molecular Neurobiology Department of Medical Biochemistry and Biophysics Scheeles väg 1, 171 77 Stockholm, Sweden<x-apple-data-detectors://1/0> +46 8 52 48 75 77<tel:+46%208%2052%2048%2075%2077> (office) +46 70 399 32 06<tel:+46%2070%20399%2032%2006> (mobile)

9 dec. 2016 kl. 17:15 skrev Job van der Zwan notifications@github.com<mailto:notifications@github.com>:

Locally, after git pulling my updates? To clarify: I only pushed the code changes to github, not to the server, because I wanted to make sure nothing was broken.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/77#issuecomment-266053205, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKag64OWJkczADiMEJKD06cW7qp3cBTks5rGX52gaJpZM4LGop8.

[JobLeonard]

Yeah, so might have been slightly broken for a moment, because that runtime-inference was part of the stuff that was lost when I lost my hard-drive. It's been re-implemented this week however, so if you pushed after that it should be fine.

[JobLeonard]

Got the schema thing working again, pushing to server

[JobLeonard]

Wait, no... I have to get the whole gcloud thing set-up again because of hte hard-drive crash... ok, it's too late for that tonight.

I pushed to git and pip, so things should work again for people who want to do stuff locally at least.

[JobLeonard]

Since this seems to have been working fine for a month now, I think we can close this bug, right?

[JobLeonard]

This is re-opened because there are are a few more points where the cache goes out of sync now:

• expanded files can be outdated
• client-side cached data is not updated, even if expanded files are updated.

Highly intertwined with #114.