generate_cache not working?

[gid000]

Describe the bug

Sorry, but I'm new here, not sure if this is the right place.

I can't find a generated cache file from a lua script command:

image:generate_cache(true, 0, 5)

After that, I'm looking at ~/.cache/darktable/mipmaps-****.d/ for .jpg

find . -iname <id>.jpg

But I can't find anything. I'm new to darktable and probably I'm missing something. I can see the script was loaded and the method I'm calling this is also executed....

Steps to reproduce

1. Create the following lua script, and load:

 local function imported(event, image)
   print("New image imported: " .. image.id)
   image:generate_cache(true, 0, 5)
end

dt.register_event("foo", "post-import-image", imported)

Expected behavior

generate a cache image?

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you install darktable from?

flatpak

darktable version

4.5.0+445.g0fcf57c4fe

What OS are you using?

Linux

What is the version of your OS?

Fedora 38

Describe your system?

No response

Are you using OpenCL GPU in darktable?

None

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[wpferguson]

image:generate_cache()isn't fast enough to be called after each image is imported. The better way is

dt.register_event(MODULE, "post-import-film", 
  function(event, images)
    for key, image in ipairs(images) do
      dt.print_log("processing image " .. key)
     image:generate_cache(true, 0, 5)
    end
  end
)

[gid000]

Hi @wpferguson , thank you.

With your suggestion, I'm seeing the print_log message (I adapted to show image.id as well):

    28.3988 LUA processing image 1:988
    29.0047 LUA processing image 2:989
    29.8370 LUA processing image 3:990
    30.4517 LUA processing image 4:991
    30.7008 LUA processing image 5:992
    30.7143 LUA processing image 6:993
    30.8330 LUA processing image 7:994
    32.2549 LUA processing image 8:995
    32.4960 LUA processing image 9:996
    32.8398 LUA processing image 10:997

However I'm getting the following error:

    33.4416 LUA ERROR : incorrect index in database
stack traceback:
    [C]: in ?
    [C]: in ?
    [C]: in metamethod 'index'
    /tmp/.mount_DarktaGjDI8D/usr/share/darktable/luarc:9: in for iterator 'for iterator'
    ./foo.lua:22: in function <./foo.lua:21>
    [C]: in ?
    [C]: in ?

Also, the cache images are still not found, I'm using (since I know ids starts with 9):

$ cd ~/.cache/darktable
$ find . -iname 9??.*

[wpferguson]

I need to see the output that was just before the error. Did it process all the images, then throw the error?

[wpferguson]

I'm confused. Are you running darktable from a flatpak or from an AppImage?

If it's a flatpak, the cache is located at ~/.var/app/org.darktable.Darktable/cache/darktable. If it's an AppImage, I have no clue where it's at (I don't even know where to get an AppImage).

It looks like you're trying to import jpg files that aren't jpgs or are corrupt in some manner. The error comes from the jpg file being in the list to import, but not being imported. When the script tries to access the file, it throws an error because the import failed.

[gid000]

hi @wpferguson sorry about the confusion. I'm running from an AppImage. AppImage, stores in a folder like this: .cache/darktable/mipmaps-3811b20baa4325315445d9dda22ca5cb0f2a45ad.d/ (unless I'm missing something). I could see some recent cache there, of images I just imported. But I can't see the ones when using lua method to get the cache.

Now, that you mention flatpak stores in a different place, this might be a problem to what I'm trying to do (I will create a new issue/question).

The error comes from the jpg file being in the list to import, but not being imported. When the script tries to access the file, it throws an error because the import failed.

Strange, because I tried to import 10 images, and all 10 are visible in my light table. Some might not be jpg indeed, but all of them are there. And what about the others that are actually JPG end returned any error? I just tried again with 10 fresh new images that are actually JPG, and I couldn't find any cache after my call.

[wpferguson]

The name of the generated cache file is mipmaps-.d/<mipmap size (0-9)/.jpg. darktable works with image ids, not original filenames.

[gid000]

The name of the generated cache file is mipmaps-.d/<mipmap size (0-9)/.jpg. darktable works with image ids, not original filenames.

I understand that. But by calling the script above, the cache is not generated. After doing a generate_cache, I still can't find them, for instance:

$ cd ~/.cache/darktable
$ find . -iname <id>.*

Where <id> is the image.id.

[wpferguson]

remember theimage:generate_cache()? That image is a data structure that has an id element, hence image.id.

Have you tried reading the API Manual? Specifically the types section.

I imported 4K+ images a couple days ago and used the Lua API image:generate_cache() function to generate the cache for all. I also did a test with JPG images and didn't have any problem there either.

[gid000]

remember the image:generate_cache()? That image is a data structure that has an id element, hence image.id.

Hi @wpferguson , I'm not asking about the image.id. I know image.id is the reference I should look at the cache folder. My steps to reproduce has an example of my script using it. And I'm also using the id during my find command. So, image.id is not my issue.

Have you tried reading the API Manual? Specifically the types section.

Yes, and unless I'm missing something, it is not clear there. That is the reason I'm asking here, stupid questions... .

For instance... your suggestion on "post-import-film", you are looping over the images argument, but according to the API 'post-import-film' uses event and film. And I can't see from the API manual how to loop over that film. In fact, I'm getting an error because of that....

Let me explain again what I'm doing:

Here is my test.lua file (as you suggested):

  1 local dt = require "darktable"
  2 
  3 dt.print_log("Loading test script...")
  4 
  5 dt.register_event(MODULE, "post-import-film",
  6   function(event, images)
  7     for key, image in ipairs(images) do
  8       dt.print_log("processing image " .. key .. ":" .. image.id)
  9       image:generate_cache(true, 5, 5)
 10     end
 11   end
 12 )

I will be testing those against those two JPG files files:

$ file ~/Downloads/23223190990_1f55a57c71_o.jpg
~/Downloads/23223190990_1f55a57c71_o.jpg: JPEG image data, Exif standard: [TIFF image data, little-endian, direntries=9, description=OLYMPUS DIGITAL CAMERA, manufacturer=OLYMPUS IMAGING CORP., model=E-M10, xresolution=174, yresolution=182, resolutionunit=2, software=Adobe Photoshop Lightroom 5.5 (Windows), datetime=2015:07:19 19:18:37], baseline, precision 8, 3290x2632, components 3

$ file ~/Downloads/41371429034_babc3f34e0_o.jpg
~/Downloads/41371429034_babc3f34e0_o.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, Exif Standard: [TIFF image data, little-endian, direntries=13, manufacturer=Canon, model=Canon EOS 1300D, orientation=upper-left, xresolution=192, yresolution=200, resolutionunit=2, software=Snapseed 2.0, datetime=2018:05:14 00:57:35], baseline, precision 8, 5184x3456, components 3

After importing the files, I can see them at darktable ighttable, so the import it worked and no error about bad JPGs.... however, my logs shows:

     1.7389 LUA Loading test script...
     9.3075 LUA processing image 1:1584
     9.5010 LUA processing image 2:1585
     9.7710 LUA ERROR : incorrect index in database
stack traceback:
    [C]: in ?
    [C]: in ?
    [C]: in metamethod 'index'
    /tmp/.mount_Darkta0An9MX/usr/share/darktable/luarc:9: in for iterator 'for iterator'
    ./test.lua:7: in function <./test.lua:6>
    [C]: in ?
    [C]: in ?

The log message suggests that I could indeed loop over the "film", but at same time is showing me a for iterator error.

After the import, I'm trying to find 1584.jpg and 1585.jpg on my cache folder, and the find command returns only cache files at the 0 folder, not the 5 (as I'm asking):

$ cd ~/.cache/darktable/mipmaps-3811b20baa4325315445d9dda22ca5cb0f2a45ad.d
$ find . -iname 1584.jpg
./0/1584.jpg

$ find . -iname 1585.jpg
./0/1585.jpg

So, darktable is generating the cache images for the 0 folder. But the lua script I provided seems to be not.

[wpferguson]

When darktable generates a cache file it remains in memory until darktable exits, at which point the file is written.

At this point I can't reproduce any of the problems with generating the cache from a lua script.

What the difference may be is that I have darktable natively installed and you're running darktable packaged as an AppImage or flatpak. Maybe you should raise your issues with the packagers.

[gid000]

When darktable generates a cache file it remains in memory until darktable exits, at which point the file is written.

That explains a lot. I just did the test, closing the app, and indeed the files are there. I would suggest adding this note to the method' API documentation. In any case, we can close this issue. Thank you.