Error running run.py

[kmsomebody]

I used prepare.py to set up TQDB. It couldn't find my TQ installation folder, but I edited the python file to just use my installation path as a hard-coded string to make it work. Everything seems to have gone fine. It went through 1906 files and said Command completed in 00:02. Then I extracted the database.arz to data/database.

Now, when I run pipenv run python ./run.py, there's an error:

C:\Programs\TQDB> pipenv run python ./run.py
03:38 Parsing locale: en
03:38 Text resource file missing: xcommonequipment.txt
03:38 Text resource file missing: x2commonequipment.txt
03:38 Text resource file missing: x3items_nonvoiced.txt
03:38 Text resource file missing: menu.txt
03:38 Text resource file missing: monsters.txt
03:38 Text resource file missing: xmonsters.txt
03:38 Text resource file missing: x2monsters.txt
03:38 Text resource file missing: x3mainquest.txt
03:38 Text resource file missing: x3mainquest_nonvoiced.txt
03:38 Text resource file missing: x3misctags_nonvoiced.txt
03:38 Text resource file missing: x3nametags_nonvoiced.txt
03:38 Text resource file missing: npc.txt
03:38 Text resource file missing: xnpc.txt
03:38 Text resource file missing: x2npc.txt
03:38 Text resource file missing: quest.txt
03:38 Text resource file missing: xquest.txt
03:38 Text resource file missing: x2quest.txt
03:38 Text resource file missing: x3servicenpcs.txt
03:38 Text resource file missing: x3sidequests.txt
03:38 Text resource file missing: x3sidequests_nonvoiced.txt
03:38 Text resource file missing: skills.txt
03:38 Text resource file missing: xskills.txt
03:38 Text resource file missing: x2skills.txt
03:38 Text resource file missing: tutorial.txt
03:38 Text resource file missing: uniqueequipment.txt
03:38 Text resource file missing: xuniqueequipment.txt
03:38 Text resource file missing: x2uniqueequipment.txt
03:38 Text resource file missing: ui.txt
03:38 Text resource file missing: xui.txt
03:38 Text resource file missing: x2ui.txt
03:38 Text resource file missing: x3basegame_nonvoiced.txt
Traceback (most recent call last):
  File "./run.py", line 109, in <module>
    tqdb()
  File "./run.py", line 92, in tqdb
    tqdb_language(args.locale)
  File "./run.py", line 32, in tqdb_language
    texts.load_locale(language)
  File "C:\Programs\TQDB\tqdb\utils\text.py", line 215, in load_locale
    self.strings['skillcostformat']
KeyError: 'skillcostformat'

[fonsleenaars]

I'm getting the same thing for #79 - apparently the ArchiveTool (or the Archives for Text/*.arc) is missing everything except commonequipment.txt. Once I solve #79 this will be solved too, it's really weird that this suddenly broke with the July 24th update.

[Sir-Mac]

@fonsleenaars The archivetool.exe is (for the version 2.8a) slightly bigger file and it is somehow broken. Using older version of the "archivetool.exe", e.g. the 2.7a version does the trick :-)

[fonsleenaars]

Thanks, I'll give that a try and maybe add it to the utils directory of this parser to make sure it doesn't happen in the future.

[IKondor]

I have same error when trying run run.py

When I tried to extract text files manually (chapter 3 of manual setup), I see that ArchiveTool.exe extract only one of 41 file:

ArchiveTool.exe Text/Text_RU.arc -extract "C:/tqdb-master/data/database"

(1 / 41) commonequipment.txt

Maybe this will help

And some question - where can I download the ArchiveTool.exe 2.7 version?

[Sir-Mac]

IKondor - Google will help you :)

[IKondor]

:)

Google helped me - ARC Explorer link With help of this little tool I have extract all files I need I extracted all files I need

But now I have new issue after run run.py:

23:20 Parsing locale: en
23:20 Parsed affixes in 0.0010025000000002393 seconds.
23:20 Parsed creatures in 0.0007972000000009416 seconds.
23:20 Parsed equipment in 0.0009866000000009478 seconds.
C:\My projects\tqdb-master\tqdb\main.py:223: FutureWarning: Possible set union at position 51 
  r'item\[(?P<index>[0-9])\](.{0,1})'
C:\My projects\tqdb-master\tqdb\main.py:223: FutureWarning: Possible set union at position 71 
  r'item\[(?P<index>[0-9])\](.{0,1})'
23:21 Parsed sets in 0.0004760000000008091 seconds.
23:21 Writing output to files...
Traceback (most recent call last):
  File "C:\Users\Олег\AppData\Roaming\Python\Python37\site-packages\PIL\ImageFile.py", line 498, in _save
    fh = fp.fileno()
AttributeError: '_idat' object has no attribute 'fileno'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 109, in <module>
    tqdb()
  File "run.py", line 95, in tqdb
    images.SpriteCreator('output/graphics/', 'output')
  File "C:\My projects\tqdb-master\tqdb\utils\images.py", line 152, in __init__
    sprite_image.save(f'{output_dir}/sprite.png', optimize=True)
  File "C:\Users\Олег\AppData\Roaming\Python\Python37\site-packages\PIL\Image.py", line 2088, in save
    save_handler(self, fp, filename)
  File "C:\Users\Олег\AppData\Roaming\Python\Python37\site-packages\PIL\PngImagePlugin.py", line 909, in _save
    ImageFile._save(im, _idat(fp, chunk), [("zip", (0, 0) + im.size, 0, rawmode)])
  File "C:\Users\Олег\AppData\Roaming\Python\Python37\site-packages\PIL\ImageFile.py", line 506, in _save
    e.setimage(im.im, b)
SystemError: tile cannot extend outside image

I have some debugging info: before program crashed, variable sprite_image in images.py (line 152) looks like this

category:0
format:None
format_description:None
height:0
im:<ImagingCore object at 0x0CF471B0>
info:{}
mode:'RGBA'
palette:None
pyaccess:None
readonly:0
size:(768, 0)
width:768
_close_exclusive_fp_after_loading:True
_size:(768, 0)

So, the image with height is 0 looks like strange :)

tq AE 2.8 steam edition win10 RU

[fonsleenaars]

I think I managed to get everything running with the latest builds, it took a bunch of manual work.

I'm going to update everything first, now that it's up and running, and see if I run into issues with other languages (very likely). Once #79 is done with this update, I'm going to update all the READMEs and everything.

I don't work in Python daily anymore, so I noticed the instructions are a little outdated and need to be a bit more cross-platform friendly. Once that's done, and I've settled on the tools to use to automate as much as possible I'll see if I can debug your particular issue, if you still have it.

[rehevkor5]

Looks like everything is working well to me. Are you still having this problem?

Some info: the AttributeError: '_idat' object has no attribute 'fileno' most likely occurred because no images were present in the output/graphics folder even after parsing all the equipment. That might be due to a problem with utils/texture/TextureViewer.exe which is used to extract the image. You could see whether you can successfully use that tool to view tex files in data/textures, or if you get some kind of error.

[chefranov]

@fonsleenaars I tried run parser and also got error. First try:

C:\tqdb>pipenv run python ./run.py --locale ru
20:42:36 INFO Parsing locale: ru
20:42:36 WARNING Text resource file missing: x2skills.txt
20:42:37 INFO NumExpr defaulting to 8 threads.
20:42:38 INFO Parsed affixes in 1.62 seconds.
20:42:41 ERROR No file found for data\database\records\skills\monster skills\melee_poison09-12_10.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\melee_poison09-12_10.dbr'
20:42:41 ERROR No file found for data\database\records\skills\monster skills\melee_poison09-12_10.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\melee_poison09-12_10.dbr'
20:42:41 ERROR No file found for data\database\records\skills\monster skills\melee_poison09-12_10.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\melee_poison09-12_10.dbr'
20:42:41 ERROR No file found for data\database\records\skills\monster skills\melee_poison09-12_10.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\melee_poison09-12_10.dbr'
20:42:41 ERROR No file found for data\database\records\skills\monster skills\melee_poison09-12_10.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\melee_poison09-12_10.dbr'
20:42:44 ERROR No file found for data\database\records\skills\monster skills\passive_buffs\retaliation_pierce01.dbr.
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 48, in read
    with open(dbr) as dbr_file:
FileNotFoundError: [Errno 2] No such file or directory: 'data\\database\\records\\skills\\monster skills\\passive_buffs\\retaliation_pierce01.dbr'
Traceback (most recent call last):
  File "C:\tqdb\tqdb\dbr.py", line 139, in parse
    prioritized_parser.parse(dbr, dbr_file, result)
  File "C:\tqdb\tqdb\parsers\equipment.py", line 161, in parse
    self.is_valid_classification(dbr, dbr_file, result)
  File "C:\tqdb\tqdb\parsers\equipment.py", line 181, in is_valid_classification
    raise InvalidItemError(f"Item {dbr_file} is excluded due to Class "
tqdb.parsers.main.InvalidItemError: Item data\database\records\creature\monster\automatoi\automatoi_heada.dbr is excluded due to Class ArmorProtective_Head with itemClassification None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./run.py", line 116, in <module>
    tqdb()
  File "./run.py", line 99, in tqdb
    tqdb_language(args.locale)
  File "./run.py", line 38, in tqdb_language
    'creatures': main.parse_creatures(),
  File "C:\tqdb\tqdb\main.py", line 219, in parse_creatures
    parsed = parse(dbr)
  File "C:\tqdb\tqdb\dbr.py", line 142, in parse
    raise InvalidItemError(f"Parser {prioritized_parser} for template "
tqdb.parsers.main.InvalidItemError: Parser <tqdb.parsers.equipment.ItemBaseParser object at 0x0FEBA5C8> for template key database\templates\templatebase\itembase.tpl tells us this item is invalid and should be ignored.

Second try:

C:\tqdb>python run.py
Traceback (most recent call last):
  File "run.py", line 8, in <module>
    from tqdb import main
  File "C:\tqdb\tqdb\main.py", line 19, in <module>
    from tqdb.utils import images
  File "C:\tqdb\tqdb\utils\images.py", line 10, in <module>
    from PIL import Image
ModuleNotFoundError: No module named 'PIL'

[fonsleenaars]

20:42:36 WARNING Text resource file missing: x2skills.txt

The russian text resources appear to be missing the skills. I'll check if that's the same for me when I get back.

[chefranov]

@fonsleenaars in second try I ran en version and there another error

C:\tqdb>python run.py
Traceback (most recent call last):
  File "run.py", line 8, in <module>
    from tqdb import main
  File "C:\tqdb\tqdb\main.py", line 19, in <module>
    from tqdb.utils import images
  File "C:\tqdb\tqdb\utils\images.py", line 10, in <module>
    from PIL import Image
ModuleNotFoundError: No module named 'PIL'

[fonsleenaars]

That would seem to indicate you haven't installed the modules through PIP yet.

You're not running it in a pipenv environment (unless you started up the virtualenv already in that above code-cap), so you would have to either

pipenv install pipenv run python run.py

Or install the modules globally (I tend to not prefer that so I can keep everything scoped per repository/project)

[rehevkor5]

I also do not see x2skills.txt in ru.

The exception when parsing the creature was due to my recent changes I think I have fixed it in https://github.com/fonsleenaars/tqdb/pull/86

The logging is a little more verbose about some of the missing things (such as skills on monsters that reference nonexistent files) than it was before, but they should not be harmful.

[chefranov]

@fonsleenaars @rehevkor5 I ran last release and it works for me, but I got a lot of errors like that: What do you think about it?

[fonsleenaars]

For some reason the skill tags are missing in the russian assets. They might be in a differently named file, or lumped with another resource, I'll take a look.

But the parser is working as expected here, there's data missing now that's the problem.

Edit:

Oh wait the errors reported are that it can't find files that are referenced, which would be what you extracted the database.arz file to. Looks like that's correct too, there's only melee_poison.dbr in that directory, they must have removed the old references and replaced them here.

[chefranov]

@fonsleenaars If skill tags missing in russian assets can we replace it from english version?

[fonsleenaars]

@fonsleenaars If skill tags missing in russian assets can we replace it from english version?

I was too hasty, see the edit in my above reply

[chefranov]

@fonsleenaars more errors

[fonsleenaars]

The errors should be "okay" in the sense that it's very common to have data in the database that's incorrectly related. There's thousands of those broken references.

If the output is still what you're looking for, you should be good to go, it just clutters the script output a bit, which we can work on silencing.

Is the output what you expect it to be, the JSON/CSS files?

[chefranov]

@fonsleenaars but it finished and generated output files It corrects data or need you check it?

[fonsleenaars]

It looked correct on my end when I spun up my website for it locally, let me know if it works for you

[chefranov]

@fonsleenaars I uploaded latest version of db on my site and it looks ok https://titanquest.org.ua/items#/ Thank you!

[fonsleenaars]

Can we close this now? Big thanks to @rehevkor5 for the fixes :)

[chefranov]

yes