eyeonus / Trade-Dangerous

Mozilla Public License 2.0
96 stars 31 forks source link

EDDBLink doesn't import listings data on Windows. On Linux it's a RAM criminal. #125

Closed Tromador closed 2 months ago

Tromador commented 2 months ago

Apparently Windows sucks. This isn't news I suppose. Following on from the great "Another process has locked the file" where the process that locked it, is in fact itself debacle, it now turns out that for some stupid reason running TD on Windows takes (on my quick box) a good 45 minutes of thrashing my spinning rust raid array, to achieve a 44MB database with no pricing data in it at all.

Linux vhosts with lots of RAM are Expensive™ to rent and my 4 core, 10GB and basically all the storage I can eat vhost is exactly the correct price at "free to an old mate". I mention this because on Linux, the import does seem to be happening, but after about 6 hours of processing is (to my best guess) half way through, has written everything to the journal (ie not made any actual commits) and using some 13GB RAM (much of which is swapped, which will be slowing it down even more).

I attach a comprehensive debug log from the windows failure, just to reassure people that no amount of -v or -w is helpful or diagnostic and that TD has decided to give no useful information about what might be the problem. Eyeonus looked at this earlier but his brain started overheating and in any case he had to go to work, so any help anyone else might be able to give would be surely appreciated. Getting it working on Windows is kinda crucial. Getting it working on Linux in a manner that takes minutes, not hours would be good too.

biglog.zip