ydataai / ydata-profiling

1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
https://docs.profiling.ydata.ai
MIT License
12.22k stars 1.65k forks source link

Report is too large for any browser to render #1579

Closed kstech-roadie closed 1 month ago

kstech-roadie commented 2 months ago

Current Behaviour

After running the profiler on ~41k records with maybe 100 or so fields, it produces an HTML report so large that no browser can render it. There was no error, and all the data is there, and all the SVGs are produced. Its just that the report tries to render as a single page. Tested with Chrome, Firefox, Brave, Edge and a couple others. They all crash because they're trying to render something like 80Mb of HTML and SVG. I would recommend breaking up the single page report into many documents that can be linked together, more like a website. Or some other technique other than shoving 80Mb into a single page. It doesnt work.

Expected Behaviour

the report is viewable

Data Description

41k records X 100 fields

Code that reproduces the bug

No response

pandas-profiling version

4.7.0

Dependencies

pandas==2.2.1

OS

macos and linux

Checklist

fabclmnt commented 1 month ago

Hi @kstech-roadie ,

thank you for opening the issue. As you can imagine the generation of a static report using HTML does have its own limitations, being one of them the amount of data and variables that can be handled through your browser.

If you want to leverage a dashboard experience that allows you to navigate bigger volumes of data, you need a different type of infrastructure and logic to support it. Give a go to YData Fabric where you can find that experience.