Closed JobLeonard closed 7 years ago
Safely creating a new folder in Python:
Implemented. On this machine it's quite a speed boost on the server side!
Ok, so I've done some more testing, using Forebrain_E9-E18.5.loom
as the test file (it's the biggest loom file online at the moment, making it a nice "if it works for this, it works for all of our loom files" kinda test-case). I downloaded the file, kept a copy of the original, then manually made it render all heatmap tiles through the client by systematically scrolling through it while the browser was at 25% zoom (this is in the order of 20 * 30 = 600 requested tiles per pan):
The benefit of doing it this way is that I immediately could see how the client handles it, and also how the server deals with many tile requests at once.
First, the resulting numbers are kind of surprising to me:
Another thing is the serving speed. It seems to be quite capable of keeping up, until a certain critical level of requests is reached, and the bottleneck is with tiles that still need to be generated. Basically, if the queue of yet-to-generate tiles hits some unknown number, it goes from milliseconds to serve a tile to 20+ seconds per tile.
I suspect that the bottleneck is writing the tiles back into the loom file, since HDF5 does not allow parallel writes.
These points together suggest to me that we're probably better off not storing the image pyramid in the loom files at all, and completely rely on saving external PNGs. We save literally ten times the storage space, avoid that the server hangs, and the loom files will be smaller to download.
We might have to change some kind of default setting so that when a loom file is uploaded to the server, all tiles are pre-generated, since it would otherwise severely screw over the first person to view a heatmap.
Just to mention that all the Loom files should come with a pre-rendered image pyramid. However, I sometimes skip that step to save time, in which case the server will need to generate them on demand.
It might make sense to have a flag on startup to force the server to render all image pyramids once and for all (by calling self.dz_get_zoom_image(0,0,8), which is the top of the pyramid, on each file).
On 6 February 2017 at 13:34:53, Job van der Zwan (notifications@github.commailto:notifications@github.com) wrote:
Ok, so I've done some more testing, using Forebrain_E9-E18.5.loom as the test file (it's the biggest loom file online at the moment, making it a nice "if it works for this, it works for all of our loom files" kinda test-case). I downloaded the file, kept a copy of the original, then manually made it render all heatmap tiles through the client by systematically scrolling through it while the browser was at 25% zoom (this is in the order of 20 * 30 = 600 requested tiles per pan):
[peek 2017-02-03 14-21small]https://cloud.githubusercontent.com/assets/259840/22646450/c49f7b84-ec6c-11e6-89b4-676644ac5933.gif
The benefit of doing it this way is that I immediately could see how the client handles it, and also how the server deals with many tile requests at once.
First, the resulting numbers are kind of surprising to me:
Another thing is the serving speed. It seems to be quite capable of keeping up, until a certain critical level of requests is reached, and the bottleneck is with tiles that still need to be generated. Basically, if the queue of yet-to-generate tiles hits some unknown number, it goes from milliseconds to serve a tile to 20+ seconds per tile.
I suspect that the bottleneck is writing the tiles back into the loom file, since HDF5 does not allow parallel writes.
These points together suggest to me that we're probably better off not storing the image pyramid in the loom files at all, and completely rely on saving external PNGs. We save literally ten times the storage space, avoid that the server hangs, and the loom files will be smaller to download.
We might have to change some kind of default setting so that when a loom file is uploaded to the server, all tiles are pre-generated, since it would otherwise severely screw over the first person to view a heatmap.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/linnarsson-lab/Loom/issues/88#issuecomment-277668372, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKag099NWnra0MAPq5WzER5lzMfwKFtks5rZxNngaJpZM4L0GSy.
Just commenting here a public sketch-pad so I keep track of how I'm reorganising the code, feel free to ignore.
Given the very significant overhead in space, time and CPU that the current scheme for serving tiles has, to the point where it can cause the server to hang, we need to reorganise it.
Here is the current order in which tiles are called:
send_tile(project, filename, z,x,y)
(loomserver.py)
dz_get_zoom_image()
, saves the returned PIL object as a PNG, and serves itLoomConnection.dz_get_zoom_image(self, x, y, z)
(loompy.py)
dz_get_zoom_tile
,LoomConnection.dz_get_zoom_tile(self, x, y, z)
(loompy.py)
Note that essentially, we have three different forms of data: the final PNG image, the PIL object, and the raw numpy array. We could convert between the PNG image and the PIL object, but in theory the conversion of the numpy array to a PIL object is lossy. In order to recursively create these tiles, we need to stay in the nparray format. It's also ridiculous that saving the PNG happens in the server code, not in the recursive tile generation code.
Here's my reorganised version that takes this into account:
LoomConnection.dz_tile_to_image(self, x, y, z, tile)
converts an nparray to a PIL (appropriating the relevant code of dz_get_zoom_image
) and returns itLoomConnection.dz_save_tile(self, x, y, z, tile)
is a new function that calls dz_tile_to_image
, then saves it as a PNG (appropriating relevant code from send_tile
for this)LoomConnection.dz_get_zoom_tile(self, x, y, z)
works like before, but just before returning the array it calls dz_save_tile
prepare_heatmap
is updated by simply replacing dz_get_zoom_image
with dz_get_zoom_tile
send_tile(project, filename, z,x,y)
checks if a PNG exists, if not it calls dz_get_zoom_tile
(after which the file should exist). Then it serves said file.LoomConnection.dz_get_zoom_image(self, x, y, z)
is removed, because it no longer serves a purposeImplemented. In theory we could make this multiprocess too, since it's an embarrasingly parallel problem, but that interferes with the threading of the server. We should really split the python code in to different parts.
In the ever growing list of "things to keep Job occupied while his laptop is taking longer to be fixed because they delivered the wrong computer part so I have to wait another week AAARGH!", I discovered this issue while testing out the website on Lars' desktop today...
Since Sten and I both use high-end machines, this problem has gone unnoticed before. Now that I'm on a slightly older desktop with a non-SSD drive and less RAM, the slowness of serving heatmap tiles becomes more obvious. We're talking 1s waiting times per tile, for tiles that are at most 8kb in size:
https://www.youtube.com/watch?v=1hvhNLxnUn0
The second point leads me to conclude that the slowdown must be happening somewhere between the lookup within the loom file and its conversion to a PNG image. The third that the main bottleneck must be the data lookup within the loom file.
One simple way to avoid this is to not look in the loom file at all: we could save the images as they are generated as tiles, and then serve those. These would function as a cache.
The idea would be as follows:
1 Given a loom file
{some name here}.loom
, create a folder called{some name here}.loom.tiles
. 2 Within this folder, tiles will be saved with the schema{z}_{x}_{y}.png
3 If a tile is requested, test if the file exists. If not, generate it 4 Serve tile from pngThis is a fairly minor change to the existing code.
Note that this is mostly for offline use of loom, for people using less-than-ultrafast-computers (I think Lars' desktop is actually quite a realistic target for users). Still, it should make the server run faster and cheaper too: serving static files is a lot faster than generating them every time a tile is served, and disk storage is probably cheaper than CPU time.
Also, the PNG files would not add much storage space: they're 8 bits per pixel, compared to 32bit floats per matrix entry, and the image tree does not add many overhead relative to the original dimensions of the matrix, since the image is quartered in size for every level we zoom out. That means the amount of points in the image tree grows as the sum of 1/k^2, which is always less than 1.645 times because:
So for a 1 GiB loom file I would expect at most around 375mib of PNG files, if all tiles were pre-generated, and ignoring that the PNGs probably compress slightly better than the loom format. Plus, and as noted above, if we generate these images lazily we save computation/storage space.
EDIT: I made a silly mistake, it's actually the sum of 1/(4^k), which is 4/3.