search

bids-standard / bids-validator

Validator for the Brain Imaging Data Structure
https://bids-standard.github.io/bids-validator/
MIT License
1 stars 4 forks source link

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit #115

Open psimonjones opened 2 days ago

psimonjones commented 2 days ago

I have been running bids-validator for a few years on an ever increasing dataset. Now I am getting

==== JS stack trace =========================================

Security context: 0x274c629257c1 1: getFiles ... and in the remaining output is listed a directory name deep inside the /derivatives directory.

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

I thought derivatives/ was not searched? It is an old version 0.26.6 so I tried version 2.0.0 using denon and through a bunary download. They both give a similar error

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

Is there a way around this memory error?

Thanks for your help

effigies commented 2 days ago

Since you were using such an old version of the legacy validator, could you try running the latest and verifying whether it was ever resolved?

npx bids-validator@1.15.0 $DATASET

And could you give the full command and output for the deno validator?

deno run -A jsr:@bids/validator@2.0 $DATASET

And how large is your dataset? Could you run the following?

find $DATASET -name derivatives -prune -o -type f -printf . | wc -c
find $DATASET/derivatives -type f -printf . | wc -c
psimonjones commented 1 day ago

Great, I installed that version and no longer get the memory error. Yesterday I installed the first thing I found on Google which was https://bids-validator.readthedocs.io/en/latest/index.html and that delivered deno version 2.0.0. But 1.15.0 does not give a memory error and is much quicker than the older version.

However, I get other bids errors now which is what I was concerned might happen if I updated it. I am reluctant to repeatedly change file contents and names every time the spec changes. I can fix most of these but I cannot stop 1: [ERR] Files with such naming scheme are not part of BIDS specification.

I tried

bids-validator $DATASET --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlink

The terminal output begins


(node:1097217) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

(node:1097217) Warning: Closing directory handle on garbage collection

              1: [ERR] Files with such naming scheme are not part of BIDS specification.

./../../../../usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

                                           Evidence: Yeo_Brainmap_fsaverage_README

And lots more Freesurfer installation examples

There is no

./../../../../usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

But there is a

/usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

Though this has nothing to do with $DATASET other than in /derivatives there are some Freesurfer results 

I then tried

cd $DATASET

bids-validator $DATASET --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlinks

and the output changed to

(node:1125058) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

(node:1125058) Warning: Closing directory handle on garbage collection

              1: [ERR] Files with such naming scheme are not part of BIDS specification.

./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

I can find no debuglog.txt 

bids-validator . --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlinks

gives the same output as above

The other questions you have

deno run -A ***@***.******@***.*** $DATASET

gives output

[2772912:0x564a8a06e000]  3626129 ms: Mark-Compact 1400.8 (1404.3) -> 1399.1 (1404.6) MB, pooled: 0 MB, 539.49 / 0.00 ms  (average mu = 0.434, current mu = 0.432) allocation failure; scavenge might not succeed

[2772912:0x564a8a06e000]  3627605 ms: Mark-Compact 1401.1 (1404.6) -> 1399.3 (1404.8) MB, pooled: 0 MB, 522.25 / 0.00 ms  (average mu = 0.564, current mu = 0.646) allocation failure; scavenge might not succeed

#

# Fatal JavaScript out of memory: Reached heap limit

#

==== C stack trace ===============================

    /home/spj24/.deno/bin/deno(+0x2d571a3) [0x564a84f0b1a3]

    /home/spj24/.deno/bin/deno(+0x2d56a6b) [0x564a84f0aa6b]

    /home/spj24/.deno/bin/deno(+0x2d51f88) [0x564a84f05f88]

    /home/spj24/.deno/bin/deno(+0x2da821c) [0x564a84f5c21c]

    /home/spj24/.deno/bin/deno(+0x2f5c867) [0x564a85110867]

    /home/spj24/.deno/bin/deno(+0x2f5aa13) [0x564a8510ea13]

    /home/spj24/.deno/bin/deno(+0x2f51611) [0x564a85105611]

    /home/spj24/.deno/bin/deno(+0x2f520e2) [0x564a851060e2]

    /home/spj24/.deno/bin/deno(+0x2f3795a) [0x564a850eb95a]

    /home/spj24/.deno/bin/deno(+0x33ae0e2) [0x564a855620e2]

    /home/spj24/.deno/bin/deno(+0x4239089) [0x564a863ed089]

Trace/breakpoint trap

find $DATASET -name derivatives -prune -o -type f -printf . | wc -c
43030

find $DATASET/derivatives -type f -printf . | wc -c

2839896
effigies commented 1 day ago

Trying to get the code quoting to work:

Great, I installed that version and no longer get the memory error. Yesterday I installed the first thing I found on Google which was https://bids-validator.readthedocs.io/en/latest/index.html and that delivered deno version 2.0.0. But 1.15.0 does not give a memory error and is much quicker than the older version.

However, I get other bids errors now which is what I was concerned might happen if I updated it. I am reluctant to repeatedly change file contents and names every time the spec changes. I can fix most of these but I cannot stop 1: [ERR] Files with such naming scheme are not part of BIDS specification.

I tried

bids-validator $DATASET --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlink

The terminal output begins


(node:1097217) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

(node:1097217) Warning: Closing directory handle on garbage collection

              1: [ERR] Files with such naming scheme are not part of BIDS specification.

./../../../../usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

                                           Evidence: Yeo_Brainmap_fsaverage_README

And lots more Freesurfer installation examples

There is no

./../../../../usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

But there is a

/usr/local/software/freesurfer/7.4.1/subjects/fsaverage/label/Yeo_Brainmap_fsaverage_README

Though this has nothing to do with $DATASET other than in /derivatives there are some Freesurfer results 

I then tried

cd $DATASET

bids-validator $DATASET --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlinks

and the output changed to

(node:1125058) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.

(Use `node --trace-deprecation ...` to show where the warning was created)

(node:1125058) Warning: Closing directory handle on garbage collection

              1: [ERR] Files with such naming scheme are not part of BIDS specification.

./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

                             ./nppdf32Log/debuglog.txt

                                           Evidence: C:\nppdf32Log\debuglog.txt

I can find no debuglog.txt 

bids-validator . --ignoreNiftiHeaders --ignoreSubjectConsistency –ignoreSymlinks

gives the same output as above

The other questions you have

deno run -A ***@***.******@***.*** $DATASET

gives output

[2772912:0x564a8a06e000]  3626129 ms: Mark-Compact 1400.8 (1404.3) -> 1399.1 (1404.6) MB, pooled: 0 MB, 539.49 / 0.00 ms  (average mu = 0.434, current mu = 0.432) allocation failure; scavenge might not succeed

[2772912:0x564a8a06e000]  3627605 ms: Mark-Compact 1401.1 (1404.6) -> 1399.3 (1404.8) MB, pooled: 0 MB, 522.25 / 0.00 ms  (average mu = 0.564, current mu = 0.646) allocation failure; scavenge might not succeed

#

# Fatal JavaScript out of memory: Reached heap limit

#

==== C stack trace ===============================

    /home/spj24/.deno/bin/deno(+0x2d571a3) [0x564a84f0b1a3]

    /home/spj24/.deno/bin/deno(+0x2d56a6b) [0x564a84f0aa6b]

    /home/spj24/.deno/bin/deno(+0x2d51f88) [0x564a84f05f88]

    /home/spj24/.deno/bin/deno(+0x2da821c) [0x564a84f5c21c]

    /home/spj24/.deno/bin/deno(+0x2f5c867) [0x564a85110867]

    /home/spj24/.deno/bin/deno(+0x2f5aa13) [0x564a8510ea13]

    /home/spj24/.deno/bin/deno(+0x2f51611) [0x564a85105611]

    /home/spj24/.deno/bin/deno(+0x2f520e2) [0x564a851060e2]

    /home/spj24/.deno/bin/deno(+0x2f3795a) [0x564a850eb95a]

    /home/spj24/.deno/bin/deno(+0x33ae0e2) [0x564a855620e2]

    /home/spj24/.deno/bin/deno(+0x4239089) [0x564a863ed089]

Trace/breakpoint trap

find $DATASET -name derivatives -prune -o -type f -printf . | wc -c
43030

find $DATASET/derivatives -type f -printf . | wc -c

2839896
effigies commented 1 day ago

Just to check, are those numbers expected? 43030 files in the raw dataset and 2.8M files in the derivatives directory?

We can probably prune derivatives more aggressively to avoid building such a large file tree (generally we haven't because building the tree is negligible compared to validation for the largest datasets we've tested with), but this might generally be a problem with such large datasets.

You might consider putting the derivatives next to the raw dataset, instead of inside:

top-level/
    raw-dataset/
    derivatives/
        deriv1/
        ...

Then you only validate top-level/raw-dataset/.

psimonjones commented 1 day ago

Yes, those numbers are right. It would be best if I could exclude derivatives. We already have a directory called derivatives in that location which is a kind of alternative derivatives. I’ll try to do a work around.

Thanks

From: Chris Markiewicz @.> Sent: 15 November 2024 13:41 To: bids-standard/bids-validator @.> Cc: Simon Jones @.>; Author @.> Subject: Re: [bids-standard/bids-validator] Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit (Issue #115)

Just to check, are those numbers expected? 43030 files in the raw dataset and 2.8M files in the derivatives directory?

We can probably prune derivatives more aggressively to avoid building such a large file tree (generally we haven't because building the tree is negligible compared to validation for the largest datasets we've tested with), but this might generally be a problem with such large datasets.

You might consider putting the derivatives next to the raw dataset, instead of inside:

top-level/ raw-dataset/ derivatives/ deriv1/ ...

Then you only validate top-level/raw-dataset/.

— Reply to this email directly, view it on GitHub https://github.com/bids-standard/bids-validator/issues/115#issuecomment-2478846893 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AFY2CCYSGVATCGU6PRAGVIT2AX2YTAVCNFSM6AAAAABR2AX23GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZYHA2DMOBZGM . You are receiving this because you authored the thread. https://github.com/notifications/beacon/AFY2CCZIHF3Z25SOTIW5DKD2AX2YTA5CNFSM6AAAAABR2AX23GWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUTYAZ22.gif Message ID: @. @.> >

psimonjones commented 1 day ago

After removing derivatives/ I am no longer getting

1: [ERR] Files with such naming scheme are not part of BIDS specification

and apparently random file names using bids-validator@1.15.0

The bids-validator also then works through

deno run -ERN jsr:@bids/validator

and the older version without the memory errors

Thanks again