search

SpaceGroupUCL / depthmapX

depthmapX is a multi-platform Spatial Network Analysis Software
197 stars 52 forks source link

Feature request: Ability to remove data from graph file to reduce file size #322

Open ugosantana opened 5 years ago

ugosantana commented 5 years ago

I know that the graph file keeps certain data to be used later on the VGA calculation. But this generates VERY BIG files. After running the Visibility Analysis, it's important to keep just the values of the Attributes. After the calculation we need the file for visualization, and not re-calculation. (I did four files in the last few days of the same building and they are 1Gb)

I don't know if I'm right, but maybe the graph file could be reduced in some way, like a PURGE command to clean up the file.

Is it possible?

pklampros commented 5 years ago

If I'm not mistaken the size of the .graph file comes from retaining the connections between the pixels. VGA creates VERY dense graphs and the more "open" your space is the heavier your file. This is required for any analysis, but I'm not sure it has to be retained or whether it can be calculated every time you want to do an analysis (or at least the first time after you open a file). That's the process that happens when you select Tools -> Visibility -> Make Visibility Graph which doesn't take very long but which also depends on the number of inter-visible pixels you have.

In the meantime, you can probably reduce the size of the .graph files by compressing them...

ugosantana commented 5 years ago

Hi Petros, I understand the concept of it: the bigger the number of pixels used (bigger space, bigger resolution of the grid...) the bigger the file. Sometimes we need to use a very small grid size. This can create very big files.

If it's possible to remove some info... I asked that because if the file contains only the "analysed" values of integration and connectivity, maybe it would be smaller... Can you test it? I don't know how easy (for me impossible) to remove this data...

pklampros commented 5 years ago

I suspect that would require a change in the .graph file format. The issue is that you can "unmake" the graph (remove the connections) but that also means you are not keeping any attributes. If we cheat and create dummy connections then older versions will not be able to read the file or will display unexpected behaviour. Thus this will have to wait for the next .graph file version which will break compatibility with any version of depthmapX that comes before it.

Did you try to compress the files by the way?

ugosantana commented 5 years ago

Well, compressing/ decompressing files is not a great thing to do regularly... but I have already tried... You have told me that already in the past. A 340m file for instance reduced to 95m. It’s a third of the size but it’s not the solution... maybe it is for a backup.

One thing you said made me realize a bigger problem: backward compatibility... That is real a problem... Anyway, if some day you guys have the time, and want to test a way to remove this data or make the new the new version read files without this data, it would be great!

I don’t even know if we would have the improvement I’m hoping for...