Command line export with Linux headless always re-imports PNG resources

[jamie-pate]

Godot version: 3.1

OS/device including version: linux_headless

Issue description:

When run with --export the headless version will re-import all resources, causing the export to take 20x longer than before.

~/bin/Godot_v3.1-stable_linux_headless.64 --export Linux/X11 Kinematic\ Character\ 3D.x86_64 
ERROR: instance: Class 'EditorSettings' can only be instantiated by editor.
   At: core/class_db.cpp:518.
ERROR: poll: /home/jpate/.config/godot/editor_settings-3.tres:3 - Parse Error: Can't create sub resource of type: EditorSettings
   At: scene/resources/resource_format_text.cpp:561.
ERROR: load: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:208.
ERROR: _load: Failed loading resource: /home/jpate/.config/godot/editor_settings-3.tres
   At: core/io/resource_loader.cpp:285.
WARNING: create: Could not open config file.
   At: editor/editor_settings.cpp:872.
reimport: begin: (Re)Importing Assets steps: 2
    reimport: step 0: purple_wood.png
    reimport: step 1: white_wood.png
savepack: begin: Packing steps: 102
    savepack: step 2: Storing File: res://cubelib.res
    savepack: step 14: Storing File: res://cubio.gdc
    savepack: step 27: Storing File: res://follow_camera.gdc
    savepack: step 39: Storing File: res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.stex
    savepack: step 39: Storing File: res://icon.png.import
    savepack: step 52: Storing File: res://level.scn
    savepack: step 64: Storing File: res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.s3tc.stex
    savepack: step 64: Storing File: res://purple_wood.png.import
    savepack: step 77: Storing File: res://purplecube.scn
    savepack: step 89: Storing File: res://.import/white_wood.png-6895acd60ce97b4315494d2be377c357.s3tc.stex
    savepack: step 89: Storing File: res://white_wood.png.import
    savepack: step 102: Storing File: res://cubio.gd.remap
    savepack: step 102: Storing File: res://follow_camera.gd.remap
    savepack: step 102: Storing File: res://icon.png
    savepack: step 102: Storing File: res://project.binary
savepack: end
reimport: end
ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

Steps to reproduce: Download attached modified version of kinematic_character demo extract

cd kinematic_character_export/kinematic_character
path/to/Godot_v3.1-stable_linux_headless.64 --export Linux/X11 Kinematic\ Character\ 3D.x86_64

Minimal reproduction project: kinematic_character_export.zip

[jamie-pate]

looks like it's getting tripped up here

Can we warn on invalid import settings? It doesn't seem to reimport them when i use the non-headless version.

editor_file_system.cpp:349

if (!ResourceFormatImporter::get_singleton()->are_import_settings_valid(p_path)) {
        //reimport settings are not valid, reimport
        return true;
}

Also, it only seems to happen for my .png files

Looks like this change: f669ebeeaf75a00b4c66dc92f62bd5030cb11660

if (!metadata.has("vram_texture")) {

but purple_wood.png.import contains

[remap]
importer="texture"
type="StreamTexture"
path.s3tc="res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.s3tc.stex"
path.etc2="res://.import/purple_wood.png-ae65a206e7a59edf759728c3bad04e56.etc2.stex"
metadata={
"imported_formats": [ "s3tc", "etc2" ],
"vram_texture": true
}

It looks like it's not recognizing path.s3tc as a valid format for the platform? ResourceFormatImporter::_get_path_and_type returns ERR_FILE_CORRUPT

VSG::storage->has_os_feature() always returns false I assume in the linux non-headless build this would work for p_feature == 's3tc'?

static bool has_server_feature_callback(const String &p_feature) {

    if (VisualServer::get_singleton()) {
        if (VisualServer::get_singleton()->has_os_feature(p_feature)) {
            return true;
        }
    }

    return false;
}

#0  RasterizerStorageDummy::has_os_feature (this=0x555559eed1b0, p_feature=...) at ./drivers/dummy/rasterizer_dummy.h:728
#1  0x0000555558176720 in VisualServerRaster::has_os_feature (this=0x555559ee3740, p_feature=...) at servers/visual/visual_server_raster.cpp:179
#2  0x000055555804c94b in has_server_feature_callback (p_feature=...) at servers/register_server_types.cpp:98
#3  0x00005555584a76c8 in OS::has_feature (this=0x7fffffffd810, p_feature=...) at core/os/os.cpp:633
#4  0x0000555558563de1 in ResourceFormatImporter::_get_path_and_type (this=0x555559e34440, p_path=..., r_path_and_type=..., r_valid=0x0) at core/io/resource_importer.cpp:85

[aaronfranke]

When you say "than before", are you comparing to Godot 3.0?

[jamie-pate]

Yes, with 3.0 I could use --export with the headless build and export each platform and it did not re-import every resource.

With 3.1 it fails to accept the resource file as up to date and re-imports every resource. (Because VSG::storage->has_os_feature() always returns false on the headless build.)

[Mithul]

Facing the same as @jamie-pate for "Linux/X11" as well. Using headless Godot Godot_v3.1.1-stable_linux_headless.64

Command ./Godot_v3.1.1-stable_linux_headless.64 --export "Linux/X11" builds/

Output

ERROR: instance: Class 'EditorSettings' can only be instantiated by editor.
   At: core/class_db.cpp:523.
ERROR: poll: /home/mithul/.config/godot/editor_settings-3.tres:3 - Parse Error: Can't create sub resource of type: EditorSettings
   At: scene/resources/resource_format_text.cpp:561.
ERROR: load: Condition ' err != OK ' is true. returned: RES()
   At: core/io/resource_loader.cpp:208.
ERROR: _load: Failed loading resource: /home/mithul/.config/godot/editor_settings-3.tres
   At: core/io/resource_loader.cpp:285.
WARNING: create: Could not open config file.
   At: editor/editor_settings.cpp:871.
ERROR: copy: Failed to open 
   At: core/os/dir_access.cpp:300.
ERROR: _fs_changed: Project export failed with error code 12.
   At: editor/editor_node.cpp:495.
ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

Using regular Linux build fails with

> ./godot --export "blah" `readlink -f builds/`
ERROR: initialize: X11 Display is not available
   At: platform/x11/os_x11.cpp:126.

The non-headless server also fails with

ERROR: _load: Method/Function Failed, returning: RES()
   At: core/io/resource_loader.cpp:285.
ERROR: start: Condition ' !scene ' is true. returned: false
   At: main/main.cpp:1739.
WARNING: cleanup: ObjectDB Instances still exist!
   At: core/object.cpp:2095.
ERROR: clear: Resources Still in use at Exit!
   At: core/resource.cpp:425.
ERROR: cleanup: Condition ' allocs_used > 0 ' is true.
   At: core/pool_vector.cpp:70.

[Mtaethefarmer]

I was able to get rid of the re-importing issue by being inside the project folder and specifying the 'project.godot' file.

godot project.godot -v -q --export "Linux/X11" ~/project/build/linux/$EXPORT_NAME.x86_64

There is still the

ERROR: ~List: Condition ' _first != __null ' is true.
   At: ./core/self_list.h:111.
ERROR: cleanup: There are still MemoryPool allocs in use at exit!
   At: core/pool_vector.cpp:70.

but that is a different issue all together. #8985

[jamie-pate]

Godot_v3.2-stable_linux_headless.64 project.godot --export Linux/X11 '../export/tmp/$EXPORTNAME-linux' does not solve the problem for me.

-q got rid of the re-import, but also failed to actually do any exporting

[Calinou]

-q got rid of the re-import

It probably just silenced the reimport as -q enables the quiet mode :wink:

[jamie-pate]

Well it also silenced the exported files. Are you sure it's not --quit?

[Calinou]

@jamie-pate --quit doesn't have a shorthand, unlike --quiet whose shorthand is -q.

[akien-mga]

--quit doesn't have a shorthand, unlike --quiet whose shorthand is -q.

Hm ;)

$ godot-git --help | egrep "\-\-(quit|quiet)"
  --quiet                          Quiet mode, silences stdout messages. Errors are still displayed.
  -q, --quit                       Quit after the first iteration.

[Calinou]

@akien-mga Seems like I was wrong then. It's kind of unexpected, -q should probably have been a shorthand for --quiet as it's a more common operation.

[fire]

Can we recreate this on the latest stable?

[realkotob]

@fire I'll try again in the weekend with the most recent 3.2 release

It was happening to me on CI for version 3.2.2 (or 3.2.3) last spring.

[jamie-pate]

still a problem on 3.3.4 when running --export-pack at least

[KoBeWi]

Is this maybe fixed in 4.0?

[jamie-pate]

https://github.com/godotengine/godot/issues/69511 maybe it's worse in godot 4? :D

Still an issue on 3.5.2-rc1 with --export-pack

$ ./test.sh 
Godot Engine v3.5.2.rc1.official.f5f0543ae - https://godotengine.org

reimport: begin: (Re)Importing Assets steps: 1
    reimport: step 0: icon-texture.png

repro-headless-png-always-reimports.zip