Open Alchemyst0x opened 2 weeks ago
Just wanted to update this issue with another discrepancy - it appears that the size calculation is very significantly off with the specific combination of flags I am using.
A bit of context: My initial interest in using the tree
command (relative to this issue) was to create JSON outputs as indexes of files for routine backups, to be stored alongside ZIP archives of the encrypted data. Since encountering the malformed JSON bug referenced in this issue, I implemented my own Python-based file tree generator with comparable JSON output for this use case. During testing, I decided to compare its results with those from tree
, particularly focusing on file sizes and the time taken to generate the output and serialize it to JSON.
Interestingly, my Python implementation consistently reported sizes that were significantly different from tree
's output, which led me to validate those size values using DaisyDisk and du
directly.
From my analysis, it turns out that the tree
command, when run with --du
in this context, was significantly overestimating the total size. I verified that my Python code produced accurate results and found that tree
over-reported the size by almost double.
Specifically, tree
reports a size of 15775465550 bytes
(roughly 15.78 GB
), while my code is accurately reporting 8719494480 bytes
(around 8.12 GB
). The Python results align with DaisyDisk's numbers, and du -sb
is reporting nearly the same bytes value, 8719418239 bytes
for this directory.
Below, you can see the respective outputs from all three tools:
# tree
$ tree -afhplDJ --du --sort mtime --dirsfirst --metafirst --prune $HOME/Downloads
# ...
{"type":"report","size":15775465550,"directories":2436,"files":19297}
# ...
# Python code
Time taken to generate and serialize file tree: 0.1827 seconds
Total: 2444 directories, 19297 files, and 0 links.
Total size: 8719494480 bytes (8.12 GB)
JSON data size: 1982222 bytes
# du
$ du -sb $HOME/Downloads
8719418239 /Users/anon/Downloads
$ du --version
du (GNU coreutils) 9.5
If I were worth a damn with C, I'd take a look myself, but C is entirely foreign to me. I'd be happy to contribute any additional information if requested, though.
Thanks again.
First, I want to say — thank you for this awesome little tool! It's a staple utility that has been incredibly helpful. Now, on to the report:
The
tree
command produces malformed JSON output when using the--du
flag in combination with the-J
(JSON output) flag. This issue occurs when attempting to access certain directories that cannot be opened, resulting in repeated or misplaced"error"
entries in the JSON structure. It can be reproduced on a macOS system when running the command from a terminal application that does not have Full Disk Access permissions.Steps to Reproduce
Minimal example demonstrating the invalid JSON output:
Output without any additional validation applied:
To clarify that this is not a filesystem-related issue, but is instead due to System Integrity Protection (SIP) restrictions on this macOS device — the above command output was produced in the VSCode terminal, an application without Full Disk Access permissions. When running the same commands from a terminal application with Full Disk Access (e.g., Kitty):
Workaround
The issue can be avoided by:
--prune
flag to exclude problematic directories.--du
flag.Additional Context
Output of
tree --version
:Output of
type -a tree
:Operating system:
The issue seems related to how
--du
calculates directory sizes and handles access errors. Adding the--prune
flag avoids the problem by excluding directories that cannot be accessed. For further context, here are the permissions as reported bystat
for the directory used in the example: