[Feature request] Add source URL to the metadata

[brunelli]

It's useful in case you wanna open the original post to read the comments, for example.

This blog post has some info about the available properties.

[brunelli]

@althonos, I've done some research today and created a fork with two commits adding the following features:

brunelli@60b67322d73f015a2a98cf27592e8cce5c1b349a:

• Add the post url to the metadata (using the XMP:Source tag).
• Add location name when available (XMP:Location).
• Add tagged users when available (XMP:PersonInImage).

brunelli@768a614ea6213061f43d5d2fe5ca103ed82fdd85:

• Add metadata to videos.

Sadly I coudn't find a Python library that allowed me to implement this, so for now I'm calling exiftool using Python's built-in subprocess.

I'd appreciate if you could take a look at the commits and maybe help me trying to find a library that we can use to write these new tags.

[disrupted]

@brunelli take a look at something I just came across https://github.com/NarrativeTeam/exifyay would this do the trick?

[brunelli]

@disrupted Unfortunately it looks like, just like Piexif, it doesn't support XMP tags (NarrativeTeam/exifyay#3).

I was taking a look at Tyf and it seems very promising. I'll do some tests tonight.

Edit: Tyf does look very powerful. Except for the lack of video metadata support. Also, it's throwing an exception trying to decode the DateTimeOriginal tag.

Edit 2: My impressions testing Tyf:

It is not reliable [in it's current state] and needs a lot of polishment. I had almost no issues getting the metadata, but my attempts to write data weren't successful. There's no documentation (yet?), so trying to work with it is a PITA.

Well, I'll keep looking for alternatives...

Edit 3-4: Well, I have photo metadata working with pyexiv2/py3exiv2 (brunelli@4e29332e47fac15c3bb813fc16e3e015502e71b7) and it looks fantastic!

For now, the only drawback is the lack of metadata writing support for videos in exiv2 (aparently it will be added in a future release).

I'll try to think of something to workaround this. Any thoughts?

I'm still calling exiftool when adding metadata to videos. It can be removed as soon as exiv2 adds writing support for mp4.

[brunelli]

Now, another matter:

To have location and tagged users, the post must be downloaded (other info is already present in the user page). InstaLooter already downloads each post if the user sets a filename template that requires more info, but currently I'm forcing it if --add-metadata is set.

I think the ideal solution would be to add keys to the --add-metadata argument.

• --add-metadata=none: no metadata will be added.

• --add-metadata or --add-metadata=basic: only download post if the filename template requires it; add available metadata.

• --add-metadata=extended: always download post and add all possible metadata.

Any thoughts on this?

[disrupted]

@brunelli thanks for your research. eager to try out your additions soon

[brunelli]

@disrupted You can test my fork with:

git clone https://github.com/brunelli/InstaLooter
cd InstaLooter
sudo pip install . --upgrade
sudo pip install py3exiv2

It'd be very helpful if you could try it and share any feedback.

[althonos]

Very nice you were able to find a pure python solution, it's definitely better than relying on an external tool.

I'd propose two different flags though, -m / --add-metadata for exif metadata and -M / --extended-metadata for XMP metadata, that would be easier to set from the CLI. Feel free to PR whenever you feel like it, if possible after adding some tests.

[brunelli]

I'd propose two different flags though, -m / --add-metadata for exif metadata and -M / --extended-metadata for XMP metadata

Just two questions:

• If the user sets both (instaLooper --add-metadata --extended-metadata), the --add-metadata arg should just be ignored, right?

• The source url doesn't require the post to be downloaded, but it is using the XMP spec. Should it always be added?

Maybe the "extended" nomenclature is not very clear. I'm starting to think of something in the line of "--add-all-metadata" or "--all-metadata".

    -m, --add-metadata           Add date, caption and url metadata to
                                 downloaded pictures (requires pyexiv2/
                                 py3exiv2) and videos (uses exiftool)
    -M, --add-all-metadata       Same as --add-metadata, but downloads each
                                 post to get all available metadata

[ghost]

@brunelli I am working from Raspbian (Debian Jessie) on a raspberryPi 2 - but I am getting error messages when trying to install py3exiv2 with pip - I think it could be a problem that I am using a pi2 which has ARM processor structure. I am not very skilled/educated regarding programming and technology and I do apologize for my english skils but i would be very thankful for any help or advise I could get since your new feature of having added even geo/loction-tags and usermentions to the pictures sounds just incredible and amazing for an archiving enthuisast :)

I now also tried pip as well as pip3 and I also did it a third time inside a newly created virtualenv (i just learnt about virtualenv two days ago..).

The error message »Datei oder Verzeichnis nicht gefunden« which is german, means »file or directory not found«. Prior trying to install py3exiv2, I run the commands you gave us above earlier on:

git clone https://github.com/brunelli/InstaLooter cd InstaLooter sudo pip install . --upgrade

then:

pi@user:~/InstaLooter $ sudo pip3 install py3exiv2
Collecting py3exiv2
  Using cached py3exiv2-0.2.1.tar.gz
Building wheels for collected packages: py3exiv2
  Running setup.py bdist_wheel for py3exiv2 ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-02r5uh27/py3exiv2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp_no6mm84pip-wheel- --python-tag cp34:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-armv7l-3.4
  creating build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/__init__.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/metadata.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/utils.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/preview.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/iptc.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/exif.py -> build/lib.linux-armv7l-3.4/pyexiv2
  copying src/pyexiv2/xmp.py -> build/lib.linux-armv7l-3.4/pyexiv2
  running build_ext
  building 'libexiv2python' extension
  creating build/temp.linux-armv7l-3.4
  creating build/temp.linux-armv7l-3.4/src
  arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -c src/exiv2wrapper.cpp -o build/temp.linux-armv7l-3.4/src/exiv2wrapper.o -g
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  In file included from src/exiv2wrapper.cpp:27:0:
  src/exiv2wrapper.hpp:32:27: fatal error: exiv2/image.hpp: Datei oder Verzeichnis nicht gefunden
   #include "exiv2/image.hpp"
                             ^
  compilation terminated.
  error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for py3exiv2
  Running setup.py clean for py3exiv2
Failed to build py3exiv2
Installing collected packages: py3exiv2
  Running setup.py install for py3exiv2 ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-02r5uh27/py3exiv2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-uc45y3w4-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.4
    creating build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/__init__.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/metadata.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/utils.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/preview.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/iptc.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/exif.py -> build/lib.linux-armv7l-3.4/pyexiv2
    copying src/pyexiv2/xmp.py -> build/lib.linux-armv7l-3.4/pyexiv2
    running build_ext
    building 'libexiv2python' extension
    creating build/temp.linux-armv7l-3.4
    creating build/temp.linux-armv7l-3.4/src
    arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.4m -c src/exiv2wrapper.cpp -o build/temp.linux-armv7l-3.4/src/exiv2wrapper.o -g
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from src/exiv2wrapper.cpp:27:0:
    src/exiv2wrapper.hpp:32:27: fatal error: exiv2/image.hpp: Datei oder Verzeichnis nicht gefunden
     #include "exiv2/image.hpp"
                               ^
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-02r5uh27/py3exiv2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-uc45y3w4-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-02r5uh27/py3exiv2/
pi@user:~/InstaLooter $ 

[brunelli]

Guys, I'm busy with college + other projects and won't be able to finish the implementation. If somebody wants to do it, check out my fork and ping me for any question.

@liorlior I think this bug doesn't belong here. Maybe try to report it in the py3exiv2 issue tracker?

[ghost]

@brunelli thank you for ythe response anyways! Yes, I will ask the py3exiv2 people.

But just to clarify, your fork does work properly and adds all the additional metadata (usertagging,geolocation..) to the downloaded pictures, right? or what are you talking about regarding work that still has to be finished on your implementation?

thanks for the patience with a beginner,

lior

[brunelli]

@liorlior Yes, everything is implemented. All that is needed is to add the command line options and write tests. Probably fix some merging conflicts too.

[ghost]

Any updates? @brunelli will you maybe work on this in The future again? Or @althonos will you implement his features e.g. adding location and people tags as metadata.