Akuukis / sepraisal

Analyse and compare Space Engineers blueprints - either uploaded or browse 100k+ steam workshop items.
https://spaceengineerspraisal.net
GNU General Public License v3.0
2 stars 4 forks source link

crawler doesn't tag hopeless steam workshop items as errored #24

Open Akuukis opened 4 years ago

Akuukis commented 4 years ago

Summary

There are couple of blueprints that have problems at steam workshop level

  1. Failure to download
  2. missing *sbc file

Steps to Reproduce actual behavior

See logs below, or run crawler yourself against those mentioned IDs

Expected behavior

For hopeless cases, tag as error'ed and don't try gain until newer script version.

Additional context

$ yarn launch 4
yarn run v1.21.1
$ yarn build && ts-node -r dotenv/config src/index.ts 4
$ yarn run tsc -v && yarn run tsc -b
$ /home/steam/sepraisal/node_modules/.bin/tsc -v
Version 3.8.3
$ /home/steam/sepraisal/node_modules/.bin/tsc -b
2020-06-09T08:54:37.572Z 'Loading routine "4-cache.ts"...'
2020-06-09T08:54:38.262Z 'Executing routine "4-cache.ts"...'
Successfully connected to server.
Required 271 blueprints.
But already cached are 259 blueprints.
But too big (>9007199254740991MB) are 0 blueprints.
Checking passwords...
WARNING: setlocale('en_US.UTF-8') failed, using locale: 'C'. International characters may not work.
Redirecting stderr to '/home/steam/.steam/logs/stderr.txt'
[  0%] Checking for available updates...
[----] Verifying installation...
Steam Console Client (c) Valve Corporation
-- type 'quit' to exit --
Loading Steam API...Warning: failed to init SDL thread priority manager: SDL not found
OK.
Logging in user '<redacted>' to Steam Public ...
Logged in OK
Waiting for user info...OK
Caching 12 blueprints...
2020-06-09T08:54:54.886Z #0    x12: !2020-06-09T08:56:00.166Z Matched an ID 1539040117 but got error: Error: Not OK but "Failure"
!2020-06-09T08:56:01.895Z Matched an ID 1403004507 but got error: Error: Not OK but "Failure"
!2020-06-09T08:56:04.019Z Matched an ID 1203707502 but got error: Error: Not OK but "Failure"
!2020-06-09T08:56:05.601Z Matched an ID 1904357579 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:07.106Z Matched an ID 1904357192 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:08.412Z Matched an ID 1904356892 but got error: Error: Unrecognized mod contents: bp.sbcB5, metadata.mod
!2020-06-09T08:56:09.448Z Matched an ID 1904355237 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:10.435Z Matched an ID 1898341252 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:11.422Z Matched an ID 1898335476 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:12.457Z Matched an ID 1898332160 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:13.444Z Matched an ID 1898329314 but got error: Error: Unrecognized mod contents: metadata.mod
!2020-06-09T08:56:14.381Z Matched an ID 1898328459 but got error: Error: Unrecognized mod contents: metadata.mod
 (0.67MB in  1m20s,    8KB/sec)
Finished in 2m37s.
Errors (0): []
Done in 101.60s.