Implement treemap profiling view for resources

[fire]

Describe the project you are working on:

3d Mobile Game

Describe how this feature / enhancement will help your project:

During optimization of the 3d game, to use over the air downloads, the entire compressed exported project must be below a specific size.

To optimize the size of the package, compare the existing filesystem layout and be able to go directly to the resource so that quality can be reduced or the texture can be shrunk.

A similiar problem is that a mobile device has limited texture memory.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

This is the truncated syntax for the Google treemap.

There can be a profiler screen with the hierarchical tree map data.

The idea is that there is an hierarchy of resources types, resource names, values and you can zoom in.

Name | Type | Size
-- | -- | --
global |   | 0
tscn | global |  
DAE | global |  
remap | global |  
fbx | global |  
no_ext | global |  
res | global |  
material | global |  
jpg | global |  
gdc | global |  
import | global |  
tres | global |  
FBX | global |  
ttf | global |  
png | global |  
Tex_Cat_Grass_Albedo.png | png | 65627422
Text_Birthday_Card.png | png | 45732392

I was not able to find a game related example of voronoi treemaps, but they're another type of treemap that is slightly easier to read than squarified treemaps.

See https://developers.google.com/chart/interactive/docs/gallery/treemap for an example of clicking, zooming and hovering for data.

Describe implementation detail for your proposal (in code), if possible:

For the resources that are exported, generate a list of resource categories, the resource name and the metric we're visualizing.

Implement a Godot profiler visualization that implements one of the squarified treemap or voronoi treemap visualizations.

Treemaps support zooming.

Clicking on a particular resource will open it.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

We worked around this by generating the needed data through CSV and using the https://developers.google.com/chart/interactive/docs/gallery/treemap API, but it is an external process.

Is there a reason why this should be core and not an add-on in the asset library?:

The core profiler cannot be extended. A treemap would be awesome at showing all the memory used. It arguably is a core profiler feature we are missing.

[Calinou]

Can the file size treemaps be created using external programs? If so, I don't think this should be in core.

Maybe someone should work on creating a treemap creation plugin or external tool :slightly_smiling_face: An external tool will also work with non-Godot projects.

[fire]

There are two parts of the problem:

1. Getting the data structure out of Godot if using an external program / accessible in Godot if using a treemap .

The texture memory / export size treemap graphic can be created with external programs, but it is not trivial to get the exported resources data structure. This is a problem too.

Yes, you can scan tscns for internal properties, but not binary resources externally.

So you can get file size treemaps, but not triangle count or any internal property.

For example I mentioned getting the file size because it's the only one able to be gotten through the external scanners, but there's also vertex count, triangle count, number of draw calls per scene, bounding scale of level of detail (lods?), or even the exact compressed texture size / texture compression type etc.

2. Visualizing the large array of properties effectively

It is not trivial to go through hundreds, thousands of properties and know their relative importance. Having a treemap interface in Godot allows the developer to process this large list of information and directly edit it.

It is possible to take the exported package and scan its .import folder and for each type of extension do the extension and file size splitting, but that does not give the ability for the developer to directly go to the texture (for example) and change its properties.

This proposal is really about making a treemap that streamlines that workflow of editing the selected resources in Godot. The integration is important.

A process of reading CSV reports converted to Google Sheets's treemaps is not really the same thing.

[Calinou]

If there's a standard format for files that can be used to generate treemaps (e.g. JSON following a specific schema), it'd be interesting to make Godot able to export it directly. Otherwise, I don't think we should implement built-in visualization as it can be a lot of work for little benefit.

[fire]

We can output csv for the google sheets format. However it's not standard. Does anyone have a standard format for treemaps?

However the main benefit is to be able to select the items that are too big in the tree map and directly delete them.

[fire]

Also found https://www.d3-graph-gallery.com/graph/treemap_json.html.

The format looks like this.

https://raw.githubusercontent.com/holtzy/D3-graph-gallery/master/DATA/data_dendrogram_full.json

[Edited]

Example of CSV to JSON:

https://github.com/d3/d3-hierarchy#stratify