[FEATURE] Test progress counter

[KITT3000]

One of my friends wanted to check the program and gave him quite a lot of assignments because he has 77GB (2227 files) of mods and the program ran for 2 hours and still did not finish work and it is not known whether it crashed or still tesed, so it would be good if there was a progress counter, e.g. on bottom of the window:

Tested 245/2227

He only saw these log entries:

NOTICE: 2021-06-22T20:18:37.067Z (collection) Finished read mods on disk (test pending)
NOTICE: 2021-06-22T20:18:38.107Z (collection) Finished read save games
FILE-ERROR: 2021-06-22T20:18:38.225Z (FS19_barbedWire_fenceKit) Zip missing file: modDesc.xml: Error: Entry not found
FILE-ERROR: 2021-06-22T20:18:38.621Z (JohnDeere_Series_8R) Zip missing file: modDesc.xml: Error: Entry not found
FILE-ERROR: 2021-06-22T20:18:38.765Z (aaa_FS19_SDC_Flatbed) Caught XML Parse error: Error: Invalid character in entity name
Line: 2
Column: 24
Char:  
FILE-ERROR: 2021-06-22T20:18:39.386Z (do tego sie kurwa nie przyznaje wiec usunalem nazwe zipa) Caught XML Parse error: Error: Malformed comment
Line: 277
Column: 22
Char: -
FILE-ERROR: 2021-06-22T20:18:39.414Z (FS19_CM_CR9000Series) Caught XML Parse error: Error: No whitespace between attributes
Line: 27
Column: 42
Char: i
FILE-ERROR: 2021-06-22T20:18:39.558Z (FS19_JohnDeere6R_2014-2018) Caught XML Parse error: Error: Invalid character in entity name
Line: 2
Column: 18
Char:  
FILE-ERROR: 2021-06-22T20:18:39.567Z (FS19_JohnDeere_6M_6R_by6195rpower) Caught XML Parse error: Error: Invalid character in entity name
Line: 2
Column: 17
Char:  
FILE-ERROR: 2021-06-22T20:18:39.592Z (FS19_kroneBiGX_Series) Caught XML Parse error: Error: Invalid character in entity name
Line: 6
Column: 34
Char:  
FILE-ERROR: 2021-06-22T20:18:39.845Z (FS19_Zetor_56xx_pack) Caught XML Parse error: Error: Invalid character in entity name
Line: 32
Column: 42
Char: "
NOTICE: 2021-06-22T20:18:40.198Z (ipcProcess) Could not get "broken list" : Error: ENOENT: no such file or directory, open

[jtsage]

This should be doable. I'll dig a bit on that log - I can tell you that the scan did error out - that "ipcProcess" at the end is it asking for the first of the lists to populate the UI and failing. But that is a catch all at the end that should have been unreachable, clearly it is not.

[jtsage]

Counter is about done. Trying to figure out how it got the that unreachable portion.

(pardon the noise, I don't want to lose my notes) Things it is not:

• key length (mod name). Valid keys can be ~134M characters long (2^27). Can't possible hit that on the filesystem.
• key value (mod name) - it's always escaped. Even if it were not, there is no character that is valid on the filesystem that isn't in a string key name.
• number of files - there is a key limit, but someone testing it 5 years ago found it to be in the neighborhood of 400k before things got wobbly. (EDIT: not this. Faked around 3,000 files. Not happy about it, but it didn't fail either. It did hang for a few seconds)

BUT

there is a memory limit. 2^32 bytes per object. I suppose it is possible we are hitting this.

Moving forward - any chance when I drop the new version your friend can re-run the scan with some sort of profiler open? (even just task manager) and look at the memory footprint - if it starts going way, way up, this would be where I look next. I'm hesitant to start here, because I'm really not sure where I could trim. I'm loading the bare basic details, and keeping nothing open.

Oh. one last bit - are any the mods in their folder overly large (maybe 2+ Gb). I don't know that my zip library in particular will cope well with that. It's not file size. I exceeded fat32 max on a file, it wasn't happy about reading it, but it did read it

Thanks!

[KITT3000]

He tested on the current version (he will also do it on the new one), here is a screan before starting the program: after starting on: and a video while working: https://youtu.be/esI61Z6RTK8

[jtsage]

This is super helpful. That drop at about 0:03 - 0:05 in usage looks a whole lot like a out-of-memory error garbage collection to me. Not sure why or where, but it gives me a bit more to go on. Thank you!

[KITT3000]

Here is a test of the new version, unfortunately the test counter does not work properly. https://user-images.githubusercontent.com/2678653/123527765-57dd3580-d6e2-11eb-940b-a3f9b60967da.mp4 log.txt

[KITT3000]

Version 1.9.25 works fine.