search

tfeldmann / organize

The file management automation tool.
http://organize.readthedocs.io
MIT License
2.31k stars 133 forks source link

Skipped files / Silent Error / Lack of verbose run #375

Open kisst opened 7 months ago

kisst commented 7 months ago

Describe the bug There is a file in the folder which should be processed but organize skipped it without any error prompt, my presumption is that the python code execution fails somewhere, but it does it so silently that I can't see where or why, and I could not find verbose or debug flag for execution only for config checking.

Screenshots

$ file 1000004517_20240324184039.jpg
1000004517_20240324184039.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 750x1000, components 3
$ organize sim organize.config.yaml  | grep 1000004517_20240324184039.jpg
$ organize check organize.config.yaml
Checking: /mypwd/organize.config.yaml
Config is valid.
$ dpkg -l | grep "organize "
ii  organize                             2.4.0-1                              all          File management automation tool

Environment (please complete the following information):

Your config file

rules:
  - name: "New images"
    subfolders: True
    locations:
      - "."
    filters:
      - extension:
        - jpg
        - jpeg
        - png
        - gif
      - python: |
          import os
          import datetime
          from PIL import Image
          from dateutil.parser import parse
          try:
              # try to read it from filename ( fn ) 
              fn_date = parse(str(path), fuzzy=True)
              year, month, day = fn_date.year, f"{fn_date.month:02d}", f"{fn_date.day:02d}"
              file_name_has_date = True
          except:
              file_name_has_date = False
          try:
              image = Image.open(path)
              exif_data = image.getexif()
              # See exif tags at https://exiv2.org/tags.html
              if 36867 in exif_data:
                  data_src = "Exif.Image.DateTimeOriginal"
                  taken_time = exif_data[36867]
                  year, month, day = taken_time.split()[0].split(":")
              elif 306 in exif_data:
                  data_src = "Exif.Image.DateTime"
                  taken_time = exif_data[306]
                  year, month, day = taken_time.split()[0].split(":")
              elif file_name_has_date:
                  data_src = "Filename"
              else:
                  data_src = "Filesystem ctime"
                  created_time = os.path.getctime(path)
                  created_time = datetime.datetime.fromtimestamp(created_time)
                  year, month, day = created_time.year, f"{created_time.month:02d}", f"{created_time.day:02d}"
              return {"year": year, "month": month, "day": day, "data_src": data_src}
          except (AttributeError, KeyError, IndexError) as exc:
              print(exc)
              # Just return the current date
              now = datetime.datetime.now()
          return {"year": now.year, "month": now.month, "day": now.day, "data_src": "FailbackToday"}
      - exif
    actions:
      - echo: "Based on {python.data_src } {path} should go to {python.year}/{python.month}/{python.day}/"
      - move:
          dest: "/main/pictures/{python.year}/{python.month}/{python.day}/"
          on_conflict: skip
kisst commented 7 months ago

Just to test I upgraded to v3.2.3 and the problem still existing there, I would be happy to try to debug it, but at this point unsure how, any pointers would be appreciated.

tfeldmann commented 7 months ago

Please update to the new v3, where your script runs fine:

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ SIMULATION                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Config: ".configs/issue375-python.yml"

⚙ Rule #0: New images ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  .venv/docx-template/docProps/thumbnail.jpeg
    - (echo) Based on Filesystem ctime .venv/docx-template/docProps/thumbnail.jpeg should go to 2024/02/16/
    - (move) Move to /main/pictures/2024/02/16/thumbnail.jpeg
  tests/resources/images-with-exif/1.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/1.jpg should go to 2018/09/29/
    - (move) Move to /main/pictures/2018/09/29/1.jpg
  tests/resources/images-with-exif/2.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/2.jpg should go to 2018/04/29/
    - (move) Move to /main/pictures/2018/04/29/2.jpg
  tests/resources/images-with-exif/3.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/3.jpg should go to 2017/08/12/
    - (move) Move to /main/pictures/2017/08/12/3.jpg
  tests/resources/images-with-exif/4.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/4.jpg should go to 2018/02/22/
    - (move) Move to /main/pictures/2018/02/22/4.jpg
  tests/resources/images-with-exif/5.jpg
    - (echo) Based on Exif.Image.DateTime tests/resources/images-with-exif/5.jpg should go to 2015/07/08/
    - (move) Move to /main/pictures/2015/07/08/5.jpg

success 6 / fail 0
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ SIMULATION                                                                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

In v3 you will also see any exceptions raised in your custom python filter.

tfeldmann commented 7 months ago

Please note the new and very much improved exif filter in v3 where you can use exiftool for parsing your images by setting an environment variable: ORGANIZE_EXIFTOOL_PATH="exiftool". Also in this version the dates and times are automatically parsed and available as datetime.datetime objects.

kisst commented 6 months ago

I tested the new exif tool filter as an option but unfortunately I couldn't find a way to reproduce the desired logic.

As per the example config above I am trying to date the images , and the method for doing it so doesn't only depends on exif data, if it exist it is the primary but if it doesn't then I check the file name too and if neither of them give me a date just use file create date.

I am still considering the add more options for example OCR from the bottom corner the date, as old photos had the date burnt in for a few years between full manual and digital cameras. I scan quite a few of those too.

I considered to just package it up into a python library and maybe add video support too , however I am way more a script kiddie rather than a programmer and writing it just for the sake of it makes no sense , unless it is used somewhere other than my own organize config .

However back to the issue , I still see skipped files and I have no means to debug the issue , any suggestions ?

kisst commented 6 months ago

Thanks for the hints so far @tfeldmann I narrowed down the error to the exif filter, here is my test setup

The folder has only a test file and a test config

/tmp/issue375$ ls -1
1ae75a1e661e44eda7e1f920d8e5e82c.jpg
test-config.yaml

The config is validated 

/tmp/issue375$ organize check test-config.yaml 
No problems found in "test-config.yaml".

Setting up and running the test config

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ organize run test-config.yaml 
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

running exiftool manually on the file in Q

/tmp/issue375$ exiftool 1ae75a1e661e44eda7e1f920d8e5e82c.jpg 
ExifTool Version Number         : 12.40
File Name                       : 1ae75a1e661e44eda7e1f920d8e5e82c.jpg
Directory                       : .
File Size                       : 111 KiB
File Modification Date/Time     : 2024:04:13 19:11:18+02:00
File Access Date/Time           : 2024:04:13 19:24:28+02:00
File Inode Change Date/Time     : 2024:04:13 19:11:18+02:00
File Permissions                : -rw-r--r--
File Type                       : JPEG
File Type Extension             : jpg
MIME Type                       : image/jpeg
JFIF Version                    : 1.01
Resolution Unit                 : None
X Resolution                    : 1
Y Resolution                    : 1
Image Width                     : 1080
Image Height                    : 1080
Encoding Process                : Progressive DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:2:0 (2 2)
Image Size                      : 1080x1080
Megapixels                      : 1.2

Also tried with full path for exiftool, same result

/tmp/issue375$ command -v exiftool 
/usr/bin/exiftool
/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="/usr/bin/exiftool"
/tmp/issue375$ organize run test-config.yaml 
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

I am on the latest release from organize and 12.40 on exiftool

/tmp/issue375$ organize --version
organize v3.2.3

/tmp/issue375$ exiftool -ver
12.40

And the config itself

---
rules:
  - name: "image with exif filter"
    subfolders: true
    locations:
      - "."
    filters:
      - extension:
          - jpg
      - exif
    actions:
      - echo: "This doesn't work"
  - name: "image without exif filter"
    subfolders: true
    locations:
      - "."
    filters:
      - extension:
          - jpg
    actions:
      - echo: "This works"

or in debug format 

Config(
    rules=[
        Rule(
            name='image with exif filter',
            enabled=True,
            targets='files',
            locations=[
                Location(
                    path=[
                        '.'
                    ],
                    min_depth=0,
                    max_depth='inherit',
                    search='breadth',
                    exclude_files=set(),
                    exclude_dirs=set(),
                    system_exclude_files={
                        '.localized',
                        '~$*',
                        'desktop.ini',
                        '.DS_Store',
                        'thumbs.db'
                    },
                    system_exclude_dirs={
                        '.git',
                        '.svn'
                    },
                    filter=None,
                    filter_dirs=None,
                    ignore_errors=False
                )
            ],
            subfolders=True,
            tags=set(),
            filters=[
                Extension(
                    extensions={
                        'jpg'
                    }
                ),
                Exif(
                    filter_tags={},
                    lowercase_keys=True
                )
            ],
            filter_mode='all',
            actions=[
                Echo(
                    msg="This doesn't work"
                )
            ]
        ),
        Rule(
            name='image without exif filter',
            enabled=True,
            targets='files',
            locations=[
                Location(
                    path=[
                        '.'
                    ],
                    min_depth=0,
                    max_depth='inherit',
                    search='breadth',
                    exclude_files=set(),
                    exclude_dirs=set(),
                    system_exclude_files={
                        '.localized',
                        '~$*',
                        'desktop.ini',
                        '.DS_Store',
                        'thumbs.db'
                    },
                    system_exclude_dirs={
                        '.git',
                        '.svn'
                    },
                    filter=None,
                    filter_dirs=None,
                    ignore_errors=False
                )
            ],
            subfolders=True,
            tags=set(),
            filters=[
                Extension(
                    extensions={
                        'jpg'
                    }
                )
            ],
            filter_mode='all',
            actions=[
                Echo(
                    msg='This works'
                )
            ]
        )
    ],
    _config_path=PosixPath('test-config.yaml')
)
tfeldmann commented 6 months ago

What is the output of echo $ORGANIZE_EXIFTOOL_PATH? I ask because your environment variable might not be set correctly. Can you try: ORGANIZE_EXIFTOOL_PATH='exiftool' organize run test-config.yaml (in a single command)?

kisst commented 6 months ago

The one liner version works fine

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH='exiftool' organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #0: image with exif filter ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This doesn't work

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 2 / fail 0

but simple variable set does not work

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="/usr/bin/exiftool"
/tmp/issue375$ echo $ORGANIZE_EXIFTOOL_PATH
/usr/bin/exiftool
organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

and

/tmp/issue375$ ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ echo $ORGANIZE_EXIFTOOL_PATH
exiftool
/tmp/issue375$ organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 1 / fail 0

The good news is if I export it then it works

/tmp/issue375$ export ORGANIZE_EXIFTOOL_PATH="exiftool"
/tmp/issue375$ organize run test-config.yaml
Config: "test-config.yaml"

⚙ Rule #0: image with exif filter ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This doesn't work

⚙ Rule #1: image without exif filter ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
./
  ./1ae75a1e661e44eda7e1f920d8e5e82c.jpg
    - (echo) This works

success 2 / fail 0

Note, if it's just set as variable then it does not show up in env

So the "fix" can, a simple documentation update to replace 

ORGANIZE_EXIFTOOL_PATH="exiftool"

to 

export ORGANIZE_EXIFTOOL_PATH="exiftool"

or alternatively the method for detecting the path could be more try to run in as a shell something like command -v exiftool, and set it based on that.

As far as my issue goes I actually do not use the exif data from organize at all, ( as described above the complex fallback logic ) so removing the filter fixed it for me, but also added the variable to .bashrc just in case I will use it at some point.

I could not find the relevant doc string in the repo, if you point me at it happy to send a PR on that, or if you consider the detection logic, where the os.environ.get default fallback value is from command -v instead of nothing, happy to attempt that path too.