{created} filter is not populated version 2.0.x

snuffop commented 2 years ago

Hi Great work on the 2.x stuff..

With the Created filter set and modified config for 2.x it seems that the created variable is not being populated

config:

  - name: "Match Images"
    locations: "~/Downloads"
    filters:
      - extension:
          - png
          - jpg
          - jpeg
          - gif
          - tiff
          - eps
      - created
    actions:
      - echo: "Found Image File CREATED: {created} {created.strftime('%Y-%m-%d')}"
      - move: "~/Nextcloud/Pictures/2020 - 2029/{created.year}/{created.month}/Daily/{created.day}/"

Output is as follows on SIM

⚙ Match Images ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/home/marty/Downloads
  Picture 2022-02-09 10-05-19.png
    - (echo) ERROR! 'None' has no attribute 'strftime'

Debug info:

│                                                                                                  │
│ /home/marty/.local/lib/python3.10/site-packages/organize/cli.py:87 in run_local                  │
│                                                                                                  │
│    84 │   │   config_dir, config_name = split(config_path)                                       │
│    85 │   │   config = open_fs(config_dir).readtext(config_name)                                 │
│    86 │   │   os.chdir(working_dir)                                                              │
│ ❱  87 │   │   core.run(rules=config, simulate=simulate)                                          │
│    88 │   except NeedsMigrationError as e:                                                       │
│    89 │   │   console.error(e, title="Config needs migration")                                   │
│    90 │   │   console.warn(                                                                      │
│ /home/marty/.local/lib/python3.10/site-packages/organize/core.py:328 in run                      │
│                                                                                                  │
│   325 │   console.summary(count)                                                                 │
│   326 │                                                                                          │
│   327 │   if count["fail"]:                                                                      │
│ ❱ 328 │   │   raise RuntimeWarning("Some actions failed.")                                       │
│   329

tfeldmann commented 2 years ago

Hey thanks a lot! On some linux systems (and depending on the local filesystem) there is no way to get the created date. The previous version of organize then just used the lastmodified time. Maybe you can try that?

But I guess I'll bring that back in the next version to not break old configs.

See https://unix.stackexchange.com/a/91200

snuffop commented 2 years ago

this is where my Python-fu is lacking. of course stat is working

🕙[ 11:31:08 ] ❯ stat Picture\ 2022-02-09\ 10-05-19.png
  File: Picture 2022-02-09 10-05-19.png
  Size: 375113          Blocks: 736        IO Block: 4096   regular file
Device: 8,2     Inode: 537744314   Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   marty)   Gid: ( 1000/   marty)
Access: 2022-02-09 10:05:19.159681249 -0500
Modify: 2022-02-09 10:05:19.559682087 -0500
Change: 2022-02-09 10:05:19.559682087 -0500
 Birth: 2022-02-09 10:05:19.159681249 -0500

how would I get that into the yaml for dynamic path?

Before the images would be sorted to /pictures/{YEAR}/{MONTH}/Daily/{DAY}

tfeldmann commented 2 years ago

So your filesystem tracks the birthtime? I guess organize should detect that by default. I'll try to get that in!

tfeldmann commented 2 years ago

I cannot reproduce on my local system. Can you do me a favor and run this config and post any entry of its output? It basically prints os.stat:

rules:
  - name: "Show file details"
    locations:
      - ~/Desktop  # or any other folder - you chose
    actions:
      - python: |
          import os

          details = fs.getinfo(fs_path, namespaces=["details"])
          print("Created:", details.created)
          print("MD Change:", details.metadata_changed)
          print("Modified:", details.modified)

          stat = os.stat(fs.getsyspath(fs_path))
          print(stat)

tfeldmann commented 2 years ago

Actually don't mind my previous comment. organize now falls back to using the stat utility 👍 Should be fixed in 2.0.9. Please try it out!

snuffop commented 2 years ago

It seems to still not populate the created var:

⚙ Match Images ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
/home/marty/Downloads
  Picture 2022-02-10 10-51-41.png
    - (echo) Found Image File CREATED: None
    - (python) Running python script.
    - (python) Created: None
    - (python) MD Change: 2022-02-10 15:51:41.696872+00:00
    - (python) Modified: 2022-02-10 15:51:41.696872+00:00
    - (python) os.stat_result(st_mode=33188, st_ino=537344899, st_dev=2050, st_nlink=1, st_uid=1000, st_gid=1000, st_size=346835, st_atime=1644508301, st_mtime=1644508301, st_ctime=1644508301)

Adding the prev comment to see output and full config of result

  - name: "Match Images"
    locations: "~/Downloads"
    enabled: true
    filters:
      - extension:
          - png
          - jpg
          - jpeg
          - gif
          - tiff
          - eps
      - created
    actions:
      - echo: "Found Image File CREATED: {created} "
      - python: |
          import os

          details = fs.getinfo(fs_path, namespaces=["details"])
          print("Created:", details.created)
          print("MD Change:", details.metadata_changed)
          print("Modified:", details.modified)

          stat = os.stat(fs.getsyspath(fs_path))
          print(stat)
      # - move: "~/Nextcloud/Pictures/2020 - 2029/{created.year}/{created.month:02}/Daily/{created.day:02}/"

Info sake

🕙[ 11:02:24 ] ❯ organize --version
organize, version 2.0.9

tfeldmann commented 2 years ago

Mh, maybe stat it behaving differently on macOS? What is the output of stat -f %B [somefile] on your system?

You can test with this rule:

rules:
  - name: "Show all file details"
    locations:
      - ~/Desktop
    actions:
      - python: |
          import subprocess

          birth = subprocess.run(
            ["stat", "-f%B", fs.getsyspath(fs_path)],
            capture_output=True,
            encoding='utf-8',
          ).stdout.strip()
          print(birth)

Edit: I just discovered that the BSD stat utility is very different from the GNU coreutils one.

tfeldmann commented 2 years ago

I added support for GNU stat. Please try with the development version:

pip3 install git+https://github.com/tfeldmann/organize.git

snuffop commented 2 years ago

Awesome! after installing the git version:

my config

  - name: "Match Images"
    locations: "~/Downloads"
    enabled: true
    filters:
      - extension:
          - png
          - jpg
          - jpeg
          - gif
          - tiff
          - eps
      - created
    actions:
      - echo: "Found Image File CREATED: {created} YEAR: {created.year} MONTH: {created.month} DAY {created.day} "

Gives me

  explosm.png
    - (echo) Found Image File CREATED: 2022-02-11 08:02:31-05:00 YEAR: 2022 MONTH: 2 DAY 11
  2022-02-11-084135_1529x1110_scrot.png
    - (echo) Found Image File CREATED: 2022-02-11 08:41:35-05:00 YEAR: 2022 MONTH: 2 DAY 11
  Picture 2022-02-10 10-51-41.png
    - (echo) Found Image File CREATED: 2022-02-10 10:51:41-05:00 YEAR: 2022 MONTH: 2 DAY 10
  2022-02-10-153140_1519x965_scrot.png
    - (echo) Found Image File CREATED: 2022-02-10 15:31:40-05:00 YEAR: 2022 MONTH: 2 DAY 10
  Xfinity.png
    - (echo) Found Image File CREATED: 2022-02-11 08:00:14-05:00 YEAR: 2022 MONTH: 2 DAY 11
  NOAA-nhc-Daily.png
    - (echo) Found Image File CREATED: 2022-02-11 08:00:21-05:00 YEAR: 2022 MONTH: 2 DAY 11
  underground.png
    - (echo) Found Image File CREATED: 2022-02-11 08:00:29-05:00 YEAR: 2022 MONTH: 2 DAY 11
  underground-famistar.png
    - (echo) Found Image File CREATED: 2022-02-11 08:00:39-05:00 YEAR: 2022 MONTH: 2 DAY 11
  weathercloud-famistar.png
    - (echo) Found Image File CREATED: 2022-02-11 08:00:50-05:00 YEAR: 2022 MONTH: 2 DAY 11
  dilbert.png
    - (echo) Found Image File CREATED: 2022-02-11 08:02:27-05:00 YEAR: 2022 MONTH: 2 DAY 11
  2022-02-11-084115_2559x640_scrot.png
    - (echo) Found Image File CREATED: 2022-02-11 08:41:15-05:00 YEAR: 2022 MONTH: 2 DAY 11

YAY!!

My Previous config had created.month:02 < Padding the month and the date as well to 0 pad the variable I've tried :02 and since I notice that the error is mentioning Jinja I was digging into variable padding there

 {created.month | pad(2) }

or

{created.month.rjust(2)}

or

{created.month.zfill(2)}

aren't working. AND hopefully this helps you and your other users and i'm not Just being a PITA.

tfeldmann commented 2 years ago

Glad that it works! You issues are definitely helping, keep it up!

Have a look at the last example here. You can now use the good old strftime:

rules:
  - name: Display the creation date
    locations: "~/Documents"
    filters:
      - created
    actions:
      - echo: "ISO Format:   {created.strftime('%Y-%m-%d')}"

Edit: I added some documentation here: https://organize.readthedocs.io/en/latest/updating-from-v1/#placeholders

tfeldmann commented 2 years ago

v2.1.0 with the new created code is now officially released 👍

snuffop commented 2 years ago

yep Closed.!

  - name: "Match Images"
    locations: "~/Downloads"
    enabled: true
    filters:
      - extension:
          - png
          - jpg
          - jpeg
          - gif
          - tiff
          - eps
      - created
    actions:
      - echo: "Found Image File CREATED: {created} YEAR: {created.strftime('%Y')} MONTH: {created.strftime('%m')} DAY: {created.strftime('%d')} "
      - move: "~/Nextcloud/Pictures/2020 - 2029/{created.year}/{created.strftime('%m')}/Daily/{created.strftime('%d')}/"

now has the expected output of

  dilbert.png
    - (echo) Found Image File CREATED: 2022-02-11 08:02:27-05:00 YEAR: 2022 MONTH: 02 DAY: 11
    - (move) Move to /home/marty/Nextcloud/Pictures/2020 - 2029/2022/02/Daily/11/dilbert.png
  2022-02-11-084115_2559x640_scrot.png
    - (echo) Found Image File CREATED: 2022-02-11 08:41:15-05:00 YEAR: 2022 MONTH: 02 DAY: 11
    - (move) Move to /home/marty/Nextcloud/Pictures/2020 - 2029/2022/02/Daily/11/2022-02-11-084115_2559x640_scrot.png

tfeldmann commented 2 years ago

👍 Just for your information - you can shorten your move command:

      - move: "~/Nextcloud/Pictures/2020 - 2029/{created.strftime('%Y/%m/Daily/%d')}/"

But I guess yours is more readable!

snuffop commented 2 years ago

good to know and Right as it's formatting the string with vars.. good call.

tfeldmann / organize

{created} filter is not populated version 2.0.x #177