Closed doggy closed 6 months ago
emmercm
commented
7 months ago igir is using graceful-fs to prevent errors like these, but clearly, it isn't catching all of them. And unfortunately, that stack trace doesn't contain much information to go off of. I'll try to spot places that this can happen.
emmercm
commented
7 months ago @doggy if you reduce the number of reader threads with something like --reader-threads 5 does that solve the issue?
doggy
commented
7 months ago Thanks your reply.
And adjust reader threads number seems not help with this even I set number to only 1 (log attached at below)
For now, I pickup all AP related rom folder to a separate folder "No-Intro-AP", then use my igir_pocket_sync.sh, the error disappears and igir can properly scan© ROMs to AP512(the TF card drive name I called).
To re-appear this issue, I moved all other ROM folders to "No-Intro-AP" and below is the result. It looks igir(current version) cannot handle all of the ROM subfolders downloaded at archive's No-Intro page.
205.9GiB of 55 games, 55 parents parsed
[19:25:13.628] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|Non-Game/MP3 CD (Parent-Clone) (20230704-115209).dat:
attempting to parse 2.3KiB of XML
[19:25:13.628] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|Non-Game/MP3 CD (Parent-Clone) (20230704-115209).dat:
parsed XML, deserializing to DAT
[19:25:13.628] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|Non-Game/MP3 CD (Parent-Clone) (20230704-115209).dat:
4.3GiB of 6 games, 6 parents parsed
[19:25:13.630] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|index.txt: attempting to parse 18.2KiB of XML
[19:25:13.630] DEBUG: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|index.txt: failed to parse DAT XML: Non-whitespace
before first tag., Line: 0, Column: 1, Char: A
[19:25:13.631] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|index.txt: attempting to parse SMDB
[19:25:13.636] TRACE: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|index.txt: failed to parse SMDB, file has no rows
[19:25:13.636] DEBUG: DATScanner: /Volumes/DockCase/_ROMs/No-Intro Love Pack (PC XML) (2024-01-14).zip|index.txt: failed to parse DAT file
[19:25:13.791] INFO: DATScanner: done scanning DAT files
✓ Scanning for DATs ·········· | 292 unique DATs found
[19:25:13.795] INFO: ROMScanner: scanning ROM files
[19:25:16.777] DEBUG: ROMScanner: found 52,312 ROM files
↻ Scanning for ROMs ·········· | ████████████████████████▒ | 52,175/52,312 | /Volumes/DockCase/_ROMs/ROMs-Sorted/No-Intro-AP/Zeebo - Zeebo/Zeebo App (Brazil)n
throw er; // Unhandled 'error' event
^
Error: EMFILE: too many open files, open '/Volumes/DockCase/_ROMs/ROMs-Sorted/No-Intro-AP/Zeebo - Zeebo/Z-Wheel (Brazil) (Es,Pt).zip'
Emitted 'error' event on ReadStream instance at:
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -24,
code: 'EMFILE',
syscall: 'open',
path: '/Volumes/DockCase/_ROMs/ROMs-Sorted/No-Intro-AP/Zeebo - Zeebo/Z-Wheel (Brazil) (Es,Pt).zip'
}
Node.js v21.5.0
➜ DockCase
➜ DockCase
➜ DockCase cat ./igir_pocket_sync.sh
#!/usr/bin/env bash
set -euo pipefail
here="$(pwd)"
# shellcheck disable=SC2064
trap "cd \"${here}\"" EXIT
cd "$(dirname "$0")"
SOURCE=/Volumes/DockCase/_ROMs/ROMs-Sorted
npx igir@latest copy extract clean -vvv \
--dat "${SOURCE}/../No-Intro*.zip" \
--dat-name-regex-exclude "/headerless/i" \
--input "${SOURCE}/No-Intro-AP/" \
--input-exclude "${SOURCE}/No-Intro/Nintendo - Game Boy Advance (e-Reader)/" \
--output "./Assets/{pocket}/common/_No-Intro/" \
--dir-letter \
`#--dir-letter-limit 1000` \
`# Leave BIOS files alone` \
--clean-exclude "./Assets/*/common/*.*" \
--overwrite-invalid \
--no-bios \
--no-bad \
--single \
--prefer-language ZH,EN,JA \
--prefer-region CHN,USA,WORLD,EUR,JPN \
--prefer-revision-newer \
--prefer-retail \
--reader-threads 1
➜ DockCase @doggy I don't think I understood your comment about what made the issue stop for you. In that case, were your input and output directories on separate disk drives?
I suspect this isn't a folder issue, but I don't know for sure.
TheBrainScrambler
commented
6 months ago I think doggy solved it by simply scanning a subset of his roms, and not all of them.
I pickup all AP related rom folder to a separate folder "No-Intro-AP", then use my igir_pocket_sync.sh, the error disappears
He made it reappear by simply moving back all his remaining roms, so in effect he was again scanning all of his roms
I moved all other ROM folders to "No-Intro-AP" and below is the result
emmercm
commented
6 months ago @doggy I think I found the issue while debugging #913, there are some file handles hanging when it isn't necessary to compute MD5 or SHA1 checksums of zip entries.
github-actions[bot]
commented
5 months ago This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Comment generated by the GitHub Lock Issues workflow.
Paste the command
Describe the bug
All ROMs fetch from archive org site. all sub-folder in ROMs-Sorted generated by script
FirstTimeCollectionSort.shbut multiple times.Expected behavior
Igir app should handle the folder which contains all my different console ROMs.
Debug logs
DAT(s) used
No-Intro Love Pack (PC XML) (2024-01-14).zip
igir version
v2.2.1
Node.js version
v21.5.0
Operating system
macOS 13.6.1 (22G313)
Additional context
tried add with
ulimit -n 1638400in the beginning of the shell file but not helps.