Data Viewer displays small floating-point numbers as 0 - Githubissues

microsoft / vscode-jupyter

VS Code Jupyter extension

https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter

MIT License

1.29k stars 292 forks source link

Data Viewer displays small floating-point numbers as 0 #11373

Open dilwong opened 2 years ago

dilwong commented 2 years ago

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and\/or Cell Scripts (.py files with #%% markers)

What happened?

Create variable in a .ipynb notebook

a = np.zeros((10,10)) + 1e-11

and show a in the Data Viewer:

wrong small number

It appears that numbers larger than or equal to 1e-10 are displayed correctly:

correct small number

but numbers smaller than 5e-11 replaced with 0. Numbers between 5e-11 and 1e-10 are rounded to 1e-10.

Is this a bug or the intended behavior? I couldn't find any information on this.

VS Code Version

1.71.1

Jupyter Extension Version

v2022.8.1002431955

Jupyter logs

No response

Coding Language and Runtime Version

Python 3.9.12

Language Extension Version (if applicable)

v2022.14.0

Anaconda Version (if applicable)

conda 4.12.0

Running Jupyter locally or remotely?

Local

vertok commented 2 years ago

plugin Excel Viewer in vscode has same behavior as described above. https://marketplace.visualstudio.com/items?itemName=GrapeCity.gc-excelviewer

the way around this issue was by setting csv-preview.formatValues to 'Never' in plugin settings. perhaps that may help here as well (here some docu from this plugin where this point was documented: https://github.com/jjuback/gc-excelviewer)