index table and data table sizes differ. post-import-image events are corrupted.

Crouze commented 2 days ago

Describe the bug

Starting a LUA script in script manager (autostyle in my case) fails with "contrib/autostyle failed to load". Starting dt with "-d lua" option shows the error message in the bug report title. If trying again for a second time it still fails to load but the rror message is different ("key 'apply your chosen style from exiftool tags' already registered for event shortcut").

Steps to reproduce

Start dt
Open script manager
Try to start autostyle
autostyle is not started and "contrib/autostyle failed to load"

Expected behavior

LUA script is started

Logfile | Screenshot | Screencast

darktable 4.8.1
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> ENABLED
  Lua                    -> ENABLED  - API version 9.3.0
  Colord                 -> ENABLED
  gPhoto2                -> ENABLED
  GMIC                   -> ENABLED  - Compressed LUTs are supported
  GraphicsMagick         -> ENABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> ENABLED
  OpenJPEG               -> ENABLED
  OpenEXR                -> ENABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

    14.3293 LUA ERROR: dtutils.lua: prequire: 223: Error loading contrib/autostyle 
    14.3294 LUA ERROR: dtutils.lua: prequire: 224: Error returned is index table and data table sizes differ.  post-import-image events are corrupted.

    14.3295 LUA ERROR: script_manager.lua: activate: 510: error loading contrib/autostyle 
    14.3296 LUA ERROR: script_manager.lua: activate: 511: error message: index table and data table sizes differ.  post-import-image events are corrupted.

   216.4282 LUA ERROR: dtutils.lua: prequire: 223: Error loading contrib/autostyle 
   216.4302 LUA ERROR: dtutils.lua: prequire: 224: Error returned is key 'apply your chosen style from exiftool tags' already registered for event shortcut  
   216.4304 LUA ERROR: script_manager.lua: activate: 510: error loading contrib/autostyle 
   216.4307 LUA ERROR: script_manager.lua: activate: 511: error message: key 'apply your chosen style from exiftool tags' already registered for event shortcut

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

4.8.1

What OS are you using?

Linux

What is the version of your OS?

Arch Linux x86_64, kernel 6.12.1-arch1-1

Describe your system?

Intel(R) Core(TM) i5-6300U (4) @ 3.00 GHz 16GB memory

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

Intel HD Graphics 520 @ 1.00 GHz [Integrated] (i915)

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

I have a second, completely separate, instance of dt on the same laptop which doesn't have this error.

The following lua scripts were started automatically:

CollectHelper
ext_editor
gimp
image_time
OpenInExplorer
rename_images

Crouze commented 2 days ago

Toggling all started lua scripts to off, restarting dt and then re-applying autostart for these lua scripts seems to have fixed it, oddly enough.

wpferguson commented 2 days ago

Could you tell me what scripts you had running when the error happened?

Error returned is index table and data table sizes differ. post-import-image events are corrupted.

This error is from the lua event registration code. Somehow it created a "partial" event, which should never happen.

error message: key 'apply your chosen style from exiftool tags' already registered for event shortcut

This is just a standard error message that you haven't configured any exif information with a style to apply.

Crouze commented 2 days ago

Could you tell me what scripts you had running when the error happened?

The above mentioned official / contrib scripts, plus the post-import-film generate_cache() script mentioned in another post:

local dt = require "darktable"

dt.register_event(MODULE, "post-import-film",
   function(event, images)
      for key, image in ipairs(images) do
         dt.print_log("generate cache " .. key .. ": " .. image.id)
         image:generate_cache(true, 2, 8)
      end
   end
)

Error returned is index table and data table sizes differ. post-import-image events are corrupted. This error is from the lua event registration code. Somehow it created a "partial" event, which should never happen.

Can/should it be cleared up, i.e. the partial event removed? Is it a leftover from something happening wrong earlier, or is the partial event created every time somehow?

error message: key 'apply your chosen style from exiftool tags' already registered for event shortcut This is just a standard error message that you haven't configured any exif information with a style to apply.

Not quite, the exif information and style to apply is set in the lua configuration. The style is applied on import.

wpferguson commented 2 days ago

Can/should it be cleared up, i.e. the partial event removed?

It should not be able to occur. That's why I'm trying to figure out how it occurred. If I can reproduce it, then I can make sure it doesn't occur or if it's not preventable then mitigate it if it does occur.

Is it a leftover from something happening wrong earlier, or is the partial event created every time somehow?

I've never seen it occur and I write/use lots of Lua scripts

Crouze commented 2 days ago

Understood. Let me know if I can be of any assistance, but as said once I cycled all started scripts the problem seems to have cleared. If there still is something lingering in a db however I would like to clean it up (I can't stand inconsistent data :wink: )

wpferguson commented 2 days ago

could you attach the generate_cache script? Just change the extension from .lua to .txt.

This error was something with script loading and event registration, so there should not be anything left laying around. When the script tries to register the event a consistency check is done before adding the event. The consistency check is what failed. I guess I could add a consistency check after an event is added to trap the culprit that actually added the bad event and then possibly revert it.

Crouze commented 2 days ago

Sure, it's a simple script. Didn't you post it elsewhere yourself?

import_generate_cache.txt

wpferguson commented 2 days ago

It looks like something I might have written, maybe when I first added generate_cache() to the API.

This and autostyle were the only scripts adding post-import-film events and neither should have caused a problem.

wpferguson commented 2 days ago

I set the priority to low because

In the 8+ years I've been using Lua scripts this is the first time I've seen it happen and I use events a lot.
The fix isn't so much a fix as it is a better way of handling the problem if it occurs. Instead of catching it the next time a script registers an event I'll move the consistency check to after the event processing so that we can throw the error but roll back the cause so that we don't break any scripts loaded after the error that use the same event.

darktable-org / darktable