"quality" settings gives a hint that my photo will be reconverted?

[vitaly-zdanevich]

But I want to upload the original quality.

[trougnouf]

This uses existing darktable export file formats / output options, the output and options are exactly the same as if you were to set the target storage to "file on disk", but only allowing what's supported on Wikimedia Commons. It works off the raw/input file and applies the processing pipeline, there is no quality loss inherent to this plugin.

Darktable does not have an option to output a copy of a (JPEG) file without going through its processing pipeline.

[vitaly-zdanevich]

My jpg was 2.1 mb, on Commons I found that it size if 2.6 mb - so it was reencoded :(

This is bad, I cannot use such great plugin because of this :(

Should I create a ticket about this in Darktable itself?

[trougnouf]

The issue you are describing is more relevant to darktable, yes.

I don't understand what is your use-case.

Any image you open through darktable will go through its processing pipeline.

Are you using darktable to develop your (ideally RAW) images? If yes then you will get exactly the same image out whether you chose "file on disk" or "Wikimedia Commons" on the darktable target storage. Otherwise, if you have JPEG images that you want to upload without doing any processing then there are other tools that do just that.

If you developed an image in darktable and saved it to disk (so that you have both the (ideally RAW) original image and the JPEG output), then make sure you are still working off the original RAW image rather than the already processed JPEG image (or darktable will process it again with whatever is in your default pipeline).

[ralfbrown]

Allowing ext=="" in register_storage_supported should enable the "copy" format to skip the reencoding.

[trougnouf]

Allowing ext=="" in register_storage_supported should enable the "copy" format to skip the reencoding.

Done in b21c2f2faa81836eca776c1404e838b876fb0c17 . I had to add an additional extension check because register_storage_supported is no longer sufficient to check if a file format is supported. For some reason exporting an image which fails that check lead to a crash (free(): double free detected in tcache 2\n Aborted (core dumped)), I don't know if that's related to darktable or the plugin. I did not test whether this works, please let me know if it's useful.

[vitaly-zdanevich]

Updated the plugin, now I see the copy. Tried to upload my photos - crash:

$ darktable -d lua
darktable 4.6.0
Copyright (C) 2012-2023 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> DISABLED
  Lua                    -> ENABLED  - API version 9.2.0
  Colord                 -> DISABLED
  gPhoto2                -> DISABLED
  GMIC                   -> DISABLED - Compressed LUTs are NOT supported
  GraphicsMagick         -> DISABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> DISABLED
  OpenJPEG               -> DISABLED
  OpenEXR                -> DISABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils.lua 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils.lua is already in search path, returning... 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/debug.lua 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/debug.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/file.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/file.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/log.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/log.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/string.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/string.lua is already in search path, returning... 
     2.4938 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/system.lua 
     2.4938 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/system.lua is already in search path, returning... 
Credentials: main-account
Performing HTTP request
  Path: https://commons.wikimedia.org/w/api.php
  Request body: action=query&format=json&meta=userinfo
  Result status:    200
  Result body:  {"batchcomplete":"","query":{"userinfo":{"id":0,"name":"188.169.244.17","anon":""}}}
Not logged in, need to login
Performing HTTP request
  Path: https://commons.wikimedia.org/w/api.php
  Request body: action=query&format=json&meta=tokens&type=login
  Result status:    200
  Result body:  {"batchcomplete":"","query":{"tokens":{"logintoken=***}}}
Performing HTTP request
  Path: https://commons.wikimedia.org/w/api.php
  Request body: action=clientlogin&loginreturnurl=https%3A%2F%2Fwww.mediawiki.org&password=***&format=json&username=Vitaly+Zdanevich&logintoken=***
  Result status:    200
  Result body:  {"clientlogin":{"status":"PASS","username":"Vitaly Zdanevich"}}
   278.7411 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:426: attempt to call a nil value (field 'ggettest')
stack traceback:
    [C]: in field 'ggettest'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:426: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:414>
PANIC: unprotected error in call to Lua API (bad argument #-1 (dt_lua_image_t expected))
Aborted

[trougnouf]

I made a crash causing typo, sorry. It's fixed in b68b1e2196e567cdefc83082a4b0176d48c01fcf

[vitaly-zdanevich]

Related on that - looks like avif is supported by MediaWiki - would you like to add that too?

[vitaly-zdanevich]

I made a crash causing typo, sorry. It's fixed in b68b1e2

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

[trougnouf]

Related on that - looks like avif is supported by MediaWiki - would you like to add that too?

It's not yet supported on Wikimedia Commons per https://phabricator.wikimedia.org/T257719

[trougnouf]

I made a crash causing typo, sorry. It's fixed in b68b1e2

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

What type of file are you trying to upload?

[vitaly-zdanevich]

xxx.JPG

[trougnouf]

xxx.JPG

I added case insensitivity in 3c4738ac4bacf1d066b1431411ee2a45517d811f

[vitaly-zdanevich]

Updated, tried again:

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

[trougnouf]

I added a fix in cc7f44647d02335e6bea1e05efebe2b8b312aead , I don't know if it will work with copy but at least the error message will be slightly more useful.

[vitaly-zdanevich]

register_storage_store: exporting the following image:
/tmp/IMG_0332.(Gonio_Fortress__red_hair_man)
    11.9038 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0335.(Gonio Fortress: red hair man sitting)
    11.9119 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0338.(Gonio Fortress: inside)
    11.9155 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0340.(Gonio Fortress: inside 2)
    11.9194 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0344.(Gonio Fortress: red hair man in entrance)
    11.9242 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0347.(Gonio Fortress: looking into hole)
    11.9277 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0389.(Gonio Fortress: red hair man staying)
    11.9325 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
    [C]: in metamethod 'concat'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
    .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
exported 7/7 images to Wikimedia Commons

[trougnouf]

I don't know why uploading image copies yields no extension. I normally get something like

register_storage_store: exporting the following image:
/tmp/DSC_4474.jpg(Trougnouf's sandbox)

image.path has the original file's extension so it could be taken from there in make_image_name and _register_storage_store_final_format_check if the temporary file's is empty.

[vitaly-zdanevich]

Hi, no insights about how to solve this?

[trougnouf]

Hi, no insights about how to solve this?

Yes, the solution is

image.path has the original file's extension so it could be taken from there in make_image_name and _register_storage_store_final_format_check if the temporary file's is empty.

I don't have time to work on it at the moment (PRs welcome)

[vitaly-zdanevich]

Thanks, I tried - but I found that image.path has full path without the extension.

[trougnouf]

Thanks, I tried - but I found that image.path has full path without the extension.

That's unfortunate. Thanks for trying. If we have no way to check that the copied file format is compatible then I guess the next best thing is to make copy an activable flag in the settings with a warning in the tooltip.

[vitaly-zdanevich]

register_storage_store() receives tmp_exp_path without file extension, so I tried

tmp_exp_path = '/tmp/' .. image.filename

In logs I see

Performing HTTP request
  Path: https://commons.wikimedia.org/w/api.php
  Request body: type=csrf&meta=tokens&action=query&format=json
  Result status:    200
  Result body:  {"batchcomplete":"","query":{"tokens":{"csrftoken=***}}}
Failed to export Gonio_Fortress__red_hair_man IMG_0332.JPG
exported 0/1 images to Wikimedia Commons

Why? Is it possible to print error details?