Closed OwenDurni closed 1 year ago
Sorry about the lateness, thanks for the report. Going to take a look at this in a bit. May be related/adjacent to this issue, or maybe I need to be more strict with the way that files are read and given to lupa (explicit encoding?).
And yeah, if you run into other issues with the load process, make some more issues for em', easier to track and resolve on my end when that's the case.
Oh, and I should probably write this down somewhere for issue contributing, but an exact mod list would be helpful for reproductions. I'm looking into putting in a special command line argument for draftsman-update
in version 2.0 that dumps all active mods/configuration/draftsman version/etc. into a single file that can be uploaded here to make it easier to file reports, but currently it has to be done manually.
Alright, I think I've found the problem. The trouble is that most of the time mod lua files are located in zip archives, which when we read from them return raw byte strings instead of performing the necessary conversion we expect. We can get around this by using io.TextIOWrapper
with the "utf-8-sig"
encoding, which according to the spec is exactly the same as "utf-8"
but is guaranteed to strip the BOM. Because for consistency's sake I want this to be guaranteed throughout the lifetime of update()
,
I've added the function archive_to_string()
and updated file_to_string()
as below:
def file_to_string(filepath):
"""
Simply grabs a file's contents and returns it as a string. Ensures that the
returned string is stripped of special unicode characters that Lupa dislikes.
"""
# "utf-8-sig" makes sure to strip BOM tokens if they exist
with open(filepath, mode="r", encoding="utf-8-sig") as file:
return file.read()
def archive_to_string(archive, filepath):
"""
Simply grabs a file with the specified name from an archive and returns it
as a string. Ensures that the returned string is stripped of special
unicode characters that Lupa dislikes.
"""
with archive.open(filepath) as target_file:
# "utf-8-sig" makes sure to strip BOM tokens if they exist
formatted = io.TextIOWrapper(target_file, encoding="utf-8-sig")
return formatted.read()
Replacing all instances of archive.read(filename)
with archive_to_string(archive, filename)
should fix this particular issue. I'm running into a few more of the errors you described, so if I can get to fixing them before I go to bed I'll make some issues to document them. This bug will certainly be fixed (and hopefully the modpack working) in version 1.0.6
.
Awesome. Thanks for the fix. It seems like you've found a solution that should be robust as well.
I've edited the first comment of this issue to include a full mod list.
Alright, I think I finally got it working. Turns out there was some very strange code like:
if ... ~= "__mod-name__/file" then
require("__mod-name__/file")
end
... which if you normalize the names to use proper paths instead of the double underscore syntax, this obviously no longer works and causes an infinite recursion. I looked everywhere for a simpler solution, but the only method I could get working required overwriting the default Lua file searcher, which was a pain. Fortunately, this means that I can now use python itself for all the file requiring, which is precisely what now is done: instead of using Lua file loading, python passes a file handle from which data is properly formatted and read, meaning that BOM encoding issues should (hopefully) never happen again under any circumstance.
I didn't do much rigorous testing on whether or not all the data is correct, but I have no real reason to doubt that it would be and would probably fall outside the scope of this particular issue anyway. A cursory glance seems convincing enough:
>>> from draftsman.data import entities
>>> entities.assembling_machines
['assembling-machine-1', 'assembling-machine-2', 'assembling-machine-3', 'oil-refinery', 'chemical-plant', 'centrifuge', 'glassworks-mk01', 'advanced-foundry-mk01', 'automated-factory-mk01', 'ground-borer', 'wpu', 'ball-mill-mk01', 'chemical-plant-mk01', 'carbon-filter', 'sand-extractor', 'classifier', 'soil-extractormk01', 'desulfurizator-unit', 'distilator', 'evaporator', 'fluid-separator', 'fts-reactor', 'gasifier', 'hpf', 'methanol-reactor', 'olefin-plant', 'jaw-crusher', 'power-house', 'quenching-tower', 'rectisol', 'solid-separator', 'tar-processing-unit', 'washer', 'glassworks-mk02', 'advanced-foundry-mk02', 'automated-factory-mk02', 'ground-borer-mk02', 'wpu-mk02', 'ball-mill-mk02', 'chemical-plant-mk02', 'carbon-filter-mk02', 'sand-extractor-mk02', 'classifier-mk02', 'soil-extractormk02', 'desulfurizator-unit-mk02', 'distilator-mk02', 'evaporator-mk02', 'fluid-separator-mk02', 'fts-reactor-mk02', 'gasifier-mk02', 'hpf-mk02', 'methanol-reactor-mk02', 'olefin-plant-mk02', 'jaw-crusher-mk02', 'power-house-mk02', 'quenching-tower-mk02', 'rectisol-mk02', 'solid-separator-mk02', 'tar-processing-unit-mk02', 'washer-mk02', 'glassworks-mk03', 'advanced-foundry-mk03', 'automated-factory-mk03', 'ground-borer-mk03', 'wpu-mk03', 'ball-mill-mk03', 'chemical-plant-mk03', 'carbon-filter-mk03', 'sand-extractor-mk03', 'classifier-mk03', 'soil-extractormk03', 'desulfurizator-unit-mk03', 'distilator-mk03', 'evaporator-mk03', 'fluid-separator-mk03', 'fts-reactor-mk03', 'gasifier-mk03', 'hpf-mk03', 'methanol-reactor-mk03', 'olefin-plant-mk03', 'jaw-crusher-mk03', 'power-house-mk03', 'quenching-tower-mk03', 'rectisol-mk03', 'solid-separator-mk03', 'tar-processing-unit-mk03', 'washer-mk03', 'glassworks-mk04', 'advanced-foundry-mk04', 'automated-factory-mk04', 'ground-borer-mk04', 'wpu-mk04', 'ball-mill-mk04', 'chemical-plant-mk04', 'carbon-filter-mk04', 'sand-extractor-mk04', 'classifier-mk04', 'soil-extractormk04', 'desulfurizator-unit-mk04', 'distilator-mk04', 'evaporator-mk04', 'fluid-separator-mk04', 'fts-reactor-mk04', 'gasifier-mk04', 'hpf-mk04', 'methanol-reactor-mk04', 'olefin-plant-mk04', 'jaw-crusher-mk04', 'power-house-mk04', 'quenching-tower-mk04', 'rectisol-mk04', 'solid-separator-mk04', 'tar-processing-unit-mk04', 'washer-mk04', 'mukmoux-pasture', 'cooling-tower-mk01', 'cooling-tower-mk02', 'agitator-mk01', 'nmf-mk01', 'secondary-crusher-mk01', 'thickener-mk01', 'gas-separator-mk01', 'hydrocyclone-mk01', 'vacuum-pump-mk01', 'automated-screener-mk01', 'centrifugal-pan-mk01', 'compressor-mk01', 'jig-mk01', 'grease-table-mk01', 'mixer-mk01', 'py-heat-exchanger', 'agitator-mk02', 'nmf-mk02', 'secondary-crusher-mk02', 'thickener-mk02', 'gas-separator-mk02', 'hydrocyclone-mk02', 'vacuum-pump-mk02', 'automated-screener-mk02', 'centrifugal-pan-mk02', 'compressor-mk02', 'jig-mk02', 'grease-table-mk02', 'mixer-mk02', 'py-heat-exchanger-mk02', 'agitator-mk03', 'nmf-mk03', 'secondary-crusher-mk03', 'thickener-mk03', 'gas-separator-mk03', 'hydrocyclone-mk03', 'vacuum-pump-mk03', 'automated-screener-mk03', 'centrifugal-pan-mk03', 'compressor-mk03', 'jig-mk03', 'grease-table-mk03', 'mixer-mk03', 'py-heat-exchanger-mk03', 'agitator-mk04', 'nmf-mk04', 'secondary-crusher-mk04', 'thickener-mk04', 'gas-separator-mk04', 'hydrocyclone-mk04', 'vacuum-pump-mk04', 'automated-screener-mk04', 'centrifugal-pan-mk04', 'compressor-mk04', 'jig-mk04', 'grease-table-mk04', 'mixer-mk04', 'py-heat-exchanger-mk04', 'fusion-reactor-mk01', 'fusion-reactor-mk02', 'centrifuge-mk01',
'neutron-absorber-mk01', 'neutron-moderator-mk01', 'nuclear-reactor-mk01', 'py-biomass-powerplant-mk01', 'py-coal-powerplant-mk01', 'py-gas-powerplant-mk01', 'py-oil-powerplant-mk01', 'centrifuge-mk02', 'neutron-absorber-mk02', 'neutron-moderator-mk02', 'nuclear-reactor-mk02', 'py-biomass-powerplant-mk02', 'py-coal-powerplant-mk02', 'py-gas-powerplant-mk02', 'py-oil-powerplant-mk02', 'centrifuge-mk03', 'neutron-absorber-mk03', 'neutron-moderator-mk03', 'nuclear-reactor-mk03', 'py-biomass-powerplant-mk03', 'py-coal-powerplant-mk03', 'py-gas-powerplant-mk03', 'py-oil-powerplant-mk03', 'centrifuge-mk04', 'neutron-absorber-mk04', 'neutron-moderator-mk04', 'nuclear-reactor-mk04', 'py-biomass-powerplant-mk04', 'py-coal-powerplant-mk04', 'py-gas-powerplant-mk04', 'py-oil-powerplant-mk04', 'lrf-building-mk01', 'solar-concentrator', 'atomizer-mk01', 'bio-printer-mk01', 'bio-reactor-mk01', 'biofactory-mk01', 'creature-chamber-mk01', 'genlab-mk01', 'incubator-mk01', 'micro-mine-mk01', 'research-center-mk01', 'slaughterhouse-mk01', 'spore-collector-mk01', 'botanical-nursery', 'rc-mk01', 'plankton-farm', 'guar-gum-plantation', 'fwf-mk01', 'cadaveric-arum-mk01', 'grods-swamp-mk01', 'kicalk-plantation-mk01', 'ralesia-plantation-mk01', 'rennea-plantation-mk01', 'sap-extractor-mk01', 'tuuphra-plantation-mk01', 'yotoi-aloe-orchard-mk01', 'bhoddos-culture-mk01', 'fawogae-plantation-mk01', 'navens-culture-mk01', 'yaedols-culture-mk01', 'arqad-hive-mk01', 'arthurian-pen-mk01', 'auog-paddock-mk01', 'cridren-enclosure-mk01', 'dingrits-pack-mk01', 'kmauts-enclosure-mk01', 'mukmoux-pasture-mk01', 'phadai-enclosure-mk01', 'phagnot-corral-mk01', 'prandium-lab-mk01', 'scrondrix-pen-mk01', 'simik-den-mk01', 'ulric-corral-mk01', 'vonix-den-mk01', 'vrauks-paddock-mk01', 'xenopen-mk01', 'zungror-lair-mk01', 'ez-ranch-mk01', 'dhilmos-pool-mk01', 'fish-farm-mk01', 'numal-reef-mk01', 'trits-reef-mk01', 'xyhiphoe-pool-mk01', 'zipir-reef-mk01', 'seaweed-crop-mk01', 'moss-farm-mk01', 'sponge-culture-mk01',
'atomizer-mk02', 'bio-printer-mk02', 'bio-reactor-mk02', 'biofactory-mk02', 'creature-chamber-mk02', 'genlab-mk02', 'incubator-mk02', 'micro-mine-mk02', 'research-center-mk02', 'slaughterhouse-mk02', 'spore-collector-mk02', 'botanical-nursery-mk02', 'rc-mk02', 'plankton-farm-mk02', 'guar-gum-plantation-mk02', 'fwf-mk02', 'cadaveric-arum-mk02', 'grods-swamp-mk02', 'kicalk-plantation-mk02', 'ralesia-plantation-mk02', 'rennea-plantation-mk02', 'sap-extractor-mk02', 'tuuphra-plantation-mk02', 'yotoi-aloe-orchard-mk02', 'bhoddos-culture-mk02', 'fawogae-plantation-mk02', 'navens-culture-mk02', 'yaedols-culture-mk02', 'arqad-hive-mk02', 'arthurian-pen-mk02', 'auog-paddock-mk02', 'cridren-enclosure-mk02', 'dingrits-pack-mk02', 'kmauts-enclosure-mk02', 'mukmoux-pasture-mk02', 'phadai-enclosure-mk02', 'phagnot-corral-mk02', 'prandium-lab-mk02', 'scrondrix-pen-mk02', 'simik-den-mk02', 'ulric-corral-mk02', 'vonix-den-mk02', 'vrauks-paddock-mk02', 'xenopen-mk02', 'zungror-lair-mk02', 'ez-ranch-mk02', 'dhilmos-pool-mk02', 'fish-farm-mk02', 'numal-reef-mk02', 'trits-reef-mk02', 'xyhiphoe-pool-mk02', 'zipir-reef-mk02', 'seaweed-crop-mk02', 'moss-farm-mk02', 'sponge-culture-mk02', 'atomizer-mk03', 'bio-printer-mk03', 'bio-reactor-mk03', 'biofactory-mk03', 'creature-chamber-mk03', 'genlab-mk03', 'incubator-mk03', 'micro-mine-mk03',
'research-center-mk03', 'slaughterhouse-mk03', 'spore-collector-mk03', 'botanical-nursery-mk03', 'rc-mk03', 'plankton-farm-mk03', 'guar-gum-plantation-mk03', 'fwf-mk03', 'cadaveric-arum-mk03', 'grods-swamp-mk03', 'kicalk-plantation-mk03', 'ralesia-plantation-mk03', 'rennea-plantation-mk03', 'sap-extractor-mk03', 'tuuphra-plantation-mk03', 'yotoi-aloe-orchard-mk03', 'bhoddos-culture-mk03', 'fawogae-plantation-mk03', 'navens-culture-mk03', 'yaedols-culture-mk03', 'arqad-hive-mk03', 'arthurian-pen-mk03', 'auog-paddock-mk03', 'cridren-enclosure-mk03', 'dingrits-pack-mk03', 'kmauts-enclosure-mk03', 'mukmoux-pasture-mk03', 'phadai-enclosure-mk03', 'phagnot-corral-mk03', 'prandium-lab-mk03', 'scrondrix-pen-mk03', 'simik-den-mk03', 'ulric-corral-mk03', 'vonix-den-mk03', 'vrauks-paddock-mk03', 'xenopen-mk03', 'zungror-lair-mk03', 'ez-ranch-mk03', 'dhilmos-pool-mk03', 'fish-farm-mk03', 'numal-reef-mk03', 'trits-reef-mk03', 'xyhiphoe-pool-mk03', 'zipir-reef-mk03', 'seaweed-crop-mk03', 'moss-farm-mk03', 'sponge-culture-mk03', 'atomizer-mk04', 'bio-printer-mk04', 'bio-reactor-mk04', 'biofactory-mk04', 'creature-chamber-mk04', 'genlab-mk04', 'incubator-mk04', 'micro-mine-mk04', 'research-center-mk04', 'slaughterhouse-mk04', 'spore-collector-mk04', 'botanical-nursery-mk04', 'rc-mk04', 'plankton-farm-mk04', 'guar-gum-plantation-mk04', 'fwf-mk04', 'cadaveric-arum-mk04', 'grods-swamp-mk04', 'kicalk-plantation-mk04', 'ralesia-plantation-mk04', 'rennea-plantation-mk04', 'sap-extractor-mk04', 'tuuphra-plantation-mk04', 'yotoi-aloe-orchard-mk04', 'bhoddos-culture-mk04', 'fawogae-plantation-mk04', 'navens-culture-mk04', 'yaedols-culture-mk04', 'arqad-hive-mk04', 'arthurian-pen-mk04', 'auog-paddock-mk04', 'cridren-enclosure-mk04', 'dingrits-pack-mk04', 'kmauts-enclosure-mk04', 'mukmoux-pasture-mk04', 'phadai-enclosure-mk04', 'phagnot-corral-mk04', 'prandium-lab-mk04', 'scrondrix-pen-mk04', 'simik-den-mk04', 'ulric-corral-mk04', 'vonix-den-mk04', 'vrauks-paddock-mk04', 'xenopen-mk04', 'ez-ranch-mk04', 'dhilmos-pool-mk04', 'fish-farm-mk04', 'numal-reef-mk04', 'trits-reef-mk04', 'xyhiphoe-pool-mk04', 'zipir-reef-mk04', 'seaweed-crop-mk04', 'moss-farm-mk04', 'sponge-culture-mk04', 'simik-boiler', 'data-array', 'antelope-enclosure-mk01', 'pyphoon-bay', 'vat-brain', 'dino-dig-site', 'clay-pit-mk01', 'chipshooter-mk01', 'pcb-factory-mk01', 'fbreactor-mk01', 'particle-accelerator-mk01', 'electronics-factory-mk01', 'pulp-mill-mk01', 'nano-assembler-mk01', 'clay-pit-mk02', 'chipshooter-mk02', 'pcb-factory-mk02', 'fbreactor-mk02', 'particle-accelerator-mk02', 'electronics-factory-mk02', 'pulp-mill-mk02', 'nano-assembler-mk02', 'clay-pit-mk03', 'chipshooter-mk03', 'pcb-factory-mk03', 'fbreactor-mk03', 'particle-accelerator-mk03', 'electronics-factory-mk03', 'pulp-mill-mk03', 'nano-assembler-mk03', 'clay-pit-mk04', 'chipshooter-mk04', 'pcb-factory-mk04', 'fbreactor-mk04', 'electronics-factory-mk04', 'pulp-mill-mk04', 'kicalk-plantation', 'moondrop-greenhouse-mk01', 'moondrop-greenhouse-mk02', 'moondrop-greenhouse-mk03', 'moondrop-greenhouse-mk04', 'zipir', 'auog-paddock', 'quantum-computer', 'coalbed-mk01', 'cracker-mk01', 'gas-refinery-mk01', 'heavy-oil-refinery-mk01', 'lor-mk01', 'pumpjack-mk01', 'reformer-mk01', 'tholin-atm-mk01', 'tholin-plant-mk01', 'upgrader-mk01', 'coalbed-mk02', 'cracker-mk02', 'gas-refinery-mk02', 'heavy-oil-refinery-mk02', 'lor-mk02', 'pumpjack-mk02', 'reformer-mk02', 'tholin-atm-mk02', 'tholin-plant-mk02', 'upgrader-mk02', 'coalbed-mk03', 'cracker-mk03', 'gas-refinery-mk03', 'heavy-oil-refinery-mk03', 'lor-mk03', 'pumpjack-mk03', 'reformer-mk03', 'tholin-atm-mk03', 'tholin-plant-mk03', 'upgrader-mk03', 'coalbed-mk04', 'cracker-mk04', 'gas-refinery-mk04', 'heavy-oil-refinery-mk04', 'lor-mk04', 'pumpjack-mk04', 'reformer-mk04', 'tholin-atm-mk04', 'tholin-plant-mk04', 'upgrader-mk04', 'fracking-rig', 'retorter', 'rhe', 'bof-mk01', 'smelter-mk01', 'casting-unit-mk01', 'eaf-mk01', 'electrolyzer-mk01', 'flotation-cell-mk01', 'hydroclassifier-mk01',
'impact-crusher-mk01', 'leaching-station-mk01', 'scrubber-mk01', 'wet-scrubber-mk01', 'bof-mk02', 'smelter-mk02', 'casting-unit-mk02', 'eaf-mk02', 'electrolyzer-mk02', 'flotation-cell-mk02', 'hydroclassifier-mk02', 'impact-crusher-mk02', 'leaching-station-mk02', 'scrubber-mk02', 'wet-scrubber-mk02', 'bof-mk03', 'smelter-mk03', 'casting-unit-mk03', 'eaf-mk03', 'electrolyzer-mk03', 'flotation-cell-mk03', 'hydroclassifier-mk03', 'impact-crusher-mk03', 'leaching-station-mk03', 'scrubber-mk03', 'wet-scrubber-mk03', 'bof-mk04', 'smelter-mk04', 'casting-unit-mk04', 'eaf-mk04', 'electrolyzer-mk04', 'flotation-cell-mk04', 'hydroclassifier-mk04', 'impact-crusher-mk04',
'leaching-station-mk04', 'scrubber-mk04', 'wet-scrubber-mk04', 'drp', 'sinter-unit', 'bitumen-seep-mk01-base', 'bitumen-seep-mk02-base', 'bitumen-seep-mk03-base', 'bitumen-seep-mk04-base', 'natural-gas-seep-mk01-base', 'tar-seep-mk01-base']
In addition, I added a prototype version of that mod-list printing command, currently prototyped as -r
or --report
:
(factorio-draftsman) > draftsman-update -p C:/Users/tfsch/AppData/Roaming/Factorio/mods --report
(zip) pyalienlifegraphics2 2.0.6
(zip) pyalienlifegraphics3 2.0.5
(zip) pyalienlifegraphics 2.0.3
(zip) pyalienlife 2.1.5
(zip) pyalternativeenergygraphics 1.0.5
(zip) pyalternativeenergy 1.2.3
(zip) pycoalprocessinggraphics 2.0.6
(zip) pycoalprocessing 2.1.2
(zip) pyfusionenergygraphics 2.0.0
(zip) pyfusionenergy 2.0.5
(zip) pyhightechgraphics 2.0.1
(zip) pyhightech 2.0.8
(zip) pyindustry 2.0.8
(zip) pypetroleumhandlinggraphics 2.0.0
(zip) pypetroleumhandling 2.1.8
(zip) pypostprocessing 0.2.8
(zip) pyraworesgraphics 2.0.1
(zip) pyrawores 2.4.8
(zip) py_mods_pack 2.0.0
(zip) stdlib 1.4.8
Gimme a bit to clean up and then I'll push this for 1.0.6
.
In case it matters this is in PyCharm IDE on Windows 10.
I was making an attempt to setup draftsman with most of the Pyanodons factorio mods installed.
Running draftsman-update resulted in the following stack trace
After some debugging, it looks like the root cause is that lupa (at least as it is currently invoked) does not handle executing when passed bytes that begin with the UTF-8 Byte Order Mark (BOM):
EF BB BF
(in hex). EF = 239The source of the BOM appears to be the
pypostprocessing_0.2.8
mod'ssettings-updates.lua
file. I had to open the file with a hex editor for the BOM to be visible.I was able to fix this by locally modifying draftsman's env.py to wrap lua file loads with the following
I'm sure there is a better way to solve the issue, but providing above as proof of concept.
P.S. That wasn't the only error, but I'll create separate issues as I make further attempts to get draftsman to update with py mods :)
Full mod list
Installed and active
AdditionalPasteSettings_1.6.4.zip Automatic_Train_Painter_1.2.0.zip AutoTrash_5.3.13.zip BottleneckLite_1.2.4.zip cybersyn_1.2.16.zip even-distribution_1.0.10.zip factoryplanner_1.1.66.zip flib_0.12.9.zip Milestones_1.3.18.zip PickerDollies_1.2.6.zip pushbutton_1.1.1.zip pyalienlifegraphics2_2.0.6.zip pyalienlifegraphics3_2.0.5.zip pyalienlifegraphics_2.0.3.zip pyalienlife_2.1.5.zip pyalternativeenergygraphics_1.0.5.zip pyalternativeenergy_1.2.3.zip pycoalprocessinggraphics_2.0.6.zip pycoalprocessing_2.1.2.zip pyfusionenergygraphics_2.0.0.zip pyfusionenergy_2.0.5.zip pyhightechgraphics_2.0.1.zip pyhightech_2.0.8.zip pyindustry_2.0.8.zip pypetroleumhandlinggraphics_2.0.0.zip pypetroleumhandling_2.1.8.zip pypostprocessing_0.2.8.zip pyraworesgraphics_2.0.1.zip pyrawores_2.4.8.zip RecipeBook_3.5.5.zip Shuttle_Train_Continued_1.1.0.zip Squeak Through_1.8.2.zip stdlib_1.4.8.zip textplates_0.6.10.zip VehicleSnap_1.18.5.zip
Installed and inactive
(alien-biomes-hr-terrain_0.6.1.zip) (alien-biomes_0.6.8.zip) (EditorExtensions_2.1.5.zip) (me playing around with modding)