transfer issue 4

[lachlan-00]

https://github.com/lachlan-00/python3-ampache/issues/4

[lachlan-00]

Hey, I was wondering: Is it possible to access custom Metadata of a song via python-ampache? I enabled bpm as a custom Metadata field. It works in the gui-interface and bpm values are displayed in the song details card. So the values must be stored somewhere in the ampache database. Do you think it might be possible to add these custom metadata fields in the output of:

playlist_songs = ampache.playlist_songs(ampache_url, ampache_api, single_playlist)

? That would be awesome :-)

[Rhythm-of-Time]

ooops... you archived the original thread. I was just trying to add a comment - I'll leave it here not to lose what I was writing. The following is about the custom Metadata implementation. Once you got the original moved, I'll copy it there... Cheers!

---

Code comment for the function update_ext_item starting at line 1549 of song.class.php mentions a song_ext_info table that I neither found in the database, nor other references to it in code. Was this from a previous version, maybe? A hook to external metadata files would be the right thing.

Now I see what you mean by confusing implementation... To me, it looks like it requires a rewrite. It's just not made for what I want to do with it... Ampache is very MP3 centric. But maybe that's ok for most users.

Logged in as an admin user, under Server Config/System (in the Edit Server Preferences dialog) I defined two Custom Metadata fields. Surprisingly, their names get written into the user_preference table, no matter whether "Apply to All" was checked. Is this systemwide or not?

All we get is two "virtual" field names in a user preference table. Where should the values be written to? Again, it is presumed they are already present in the files.

One might expect these fields to be attached to all songs (after the potentially dangerous altering of the song table structure to allow for two additional fields). But when does this happen - if at all? Verifying the catalogs does not trigger it. I have not found a function for this in the Metadata implementation.

"Tags" in Ampache could be a workaround to input custom metadata - at least there already is a table for them. Some require input validation, like the ISRC. That would be feasible on the client side, but the whole approach is not really elegant.

[lachlan-00]

I've got the fix for this @DjTuxeedo

<?xml version="1.0" encoding="UTF-8" ?>
<root>
<total_count>1</total_count>
<song id="77371">
    <title><![CDATA[Welcome to Planet Sexor]]></title>
    <name><![CDATA[Welcome to Planet Sexor]]></name>
    <artist id="4144"><![CDATA[Tiga]]></artist>
    <album id="51457"><![CDATA[Sexor]]></album>
    <tag id="4" count="1" ><![CDATA[Electronic]]></tag>
    <filename><![CDATA[/mnt/music/Tiga/(2005) Sexor/101 - Welcome to Planet Sexor.mp3]]></filename>
    <track>1</track>
    <time>50</time>
    <year>2006</year>
    <bitrate>192000</bitrate>
    <rate>44100</rate>
    <mode>cbr</mode>
    <mime>audio/mpeg</mime>
    <url><![CDATA[https://music.com.au/play/index.php?ssid=e07b04026bb95d2cc47190afa7feab6e&type=song&oid=77371&uid=2&player=api&name=Tiga%20-%20Welcome%20to%20Planet%20Sexor.mp3]]></url>
    <size>1209827</size>
    <mbid>936650f9-00e4-4476-accd-ccb5cf2c1d5b</mbid>
    <album_mbid></album_mbid>
    <artist_mbid></artist_mbid>
    <albumartist_mbid></albumartist_mbid>
    <art><![CDATA[https://music.com.au/image.php?object_id=51457&object_type=album&auth=e07b04026bb95d2cc47190afa7feab6e&name=art.jpg]]></art>
    <flag>0</flag>
    <preciserating>1</preciserating>
    <rating>1</rating>
    <averagerating>0</averagerating>
    <composer><![CDATA[]]></composer>
    <channels></channels>
    <comment><![CDATA[]]></comment>
    <publisher><![CDATA[Play It Again Sam; Different Recordings]]></publisher>
    <language></language>
    <replaygain_album_gain>0.000000</replaygain_album_gain>
    <replaygain_album_peak>0.000000</replaygain_album_peak>
    <replaygain_track_gain>-5.270000</replaygain_track_gain>
    <replaygain_track_peak>1.000000</replaygain_track_peak>
    <encoding><![CDATA[UTF-8]]></encoding>
    <disk><![CDATA[1]]></disk>
    <totaldisks><![CDATA[1]]></totaldisks>
    <band><![CDATA[Tiga]]></band>
    <release_type><![CDATA[album]]></release_type>
    <barcode><![CDATA[5413356594027]]></barcode>
    <audio_codec><![CDATA[mp3]]></audio_codec>
    <recording_time><![CDATA[2006-02-06]]></recording_time>
    <bpm><![CDATA[114]]></bpm>
    <album_artist_sort_order><![CDATA[Tiga]]></album_artist_sort_order>
    <performer_sort_order><![CDATA[Tiga]]></performer_sort_order>
    <media_type><![CDATA[CD]]></media_type>
    <original_release_time><![CDATA[2005-12]]></original_release_time>
    <totaltracks><![CDATA[14]]></totaltracks>
    <track_number><![CDATA[1]]></track_number>
    <isrc><![CDATA[BEP010500070]]></isrc>
    <tagging_time><![CDATA[2013-07-24T07:48:29]]></tagging_time>
    <genre><![CDATA[Electronic]]></genre>
</song>
</root>

@AshotN json_data will need to do the same.

            if (Song::isCustomMetadataEnabled()) {
                foreach ($song->getMetadata() as $metadata) {
                    $meta_name = $metadata->getField()->getName();
                    $string .= "\t<" . $meta_name . "><![CDATA[" . $metadata->getData() . "]]></" . $meta_name . ">\n";
                }
            }

[lachlan-00]

which i think is accomplished by this change

[lachlan-00]

That's valid json, @DjTuxeedo i think this covers you completely

[
    {
        "id": "77371",
        "title": "Welcome to Planet Sexor",
        "artist": {
            "id": "4144",
            "name": "Tiga"
        },
        "album": {
            "id": "51457",
            "name": "Sexor"
        },
        "filename": "\/mnt\/music\/Tiga\/(2005) Sexor\/101 - Welcome to Planet Sexor.mp3",
        "track": "1",
        "playlisttrack": "",
        "time": 50,
        "year": "2006",
        "bitrate": "192000",
        "rate": "44100",
        "mode": "cbr",
        "mime": "audio\/mpeg",
        "url": "https:\/\/music.lachlandewaard.org\/play\/index.php?ssid=e07b04026bb95d2cc47190afa7feab6e&type=song&oid=77371&uid=2&player=api&name=Tiga%20-%20Welcome%20to%20Planet%20Sexor.mp3",
        "size": "1209827",
        "mbid": "936650f9-00e4-4476-accd-ccb5cf2c1d5b",
        "album_mbid": null,
        "artist_mbid": null,
        "albumartist_mbid": null,
        "art": "https:\/\/music.lachlandewaard.org\/image.php?object_id=51457&object_type=album&auth=e07b04026bb95d2cc47190afa7feab6e&name=art.jpg",
        "flag": 0,
        "preciserating": 1,
        "rating": 1,
        "averagerating": 0,
        "composer": "",
        "channels": null,
        "comment": "",
        "publisher": "Play It Again Sam; Different Recordings",
        "language": "",
        "replaygain_album_gain": "0.000000",
        "replaygain_album_peak": "0.000000",
        "replaygain_track_gain": "-5.270000",
        "replaygain_track_peak": "1.000000",
        "encoding": "UTF-8",
        "disk": "1",
        "totaldisks": "1",
        "band": "Tiga",
        "release_type": "album",
        "barcode": "5413356594027",
        "audio_codec": "mp3",
        "recording_time": "2006-02-06",
        "bpm": "114",
        "album_artist_sort_order": "Tiga",
        "performer_sort_order": "Tiga",
        "media_type": "CD",
        "original_release_time": "2005-12",
        "totaltracks": "14",
        "track_number": "1",
        "isrc": "BEP010500070",
        "tagging_time": "2013-07-24T07:48:29",
        "tags": [
            "Electronic"
        ]
    }
]

[lachlan-00]

I forgot I need to put a browse/search filter too

[Rhythm-of-Time]

What is really the difference between metadata, custom metadata and tags? Are tags always Genres? They do have their own type "tag", while metadata are text and custom metadata are "mixed". The mixed type means what exactly?

Call me a dummy, but I only partly understand the docs on search here - https://github.com/ampache/ampache/wiki/XML-methods#advanced_search could you please post an example for a rules array with three rules using "and | or", please?

[lachlan-00]

Sure, metadata is the only mixed type whish isn't great as it adds another search array. I only just worked out what mixed was when we started on this.

I'm spending my day at home tomorrow so I'll be working on a few ampache things and will put up some examples. The mixed types actually need another writeup and the current code for python doesn't support the additional rule required.

In the docs folder here I think example.py uses a regular advanced search command as well

[lachlan-00]

to make a general advanced_search in web you can make the rules explicitly

http://music.com.au/server/xml.server.php?action=advanced_search&auth=AUTHKEY&version=400001&rule_1=year&rule_1_operator=2&rule_1_input=1999

In python you just make the rules in the correct order.

# inputs  [rule_1, rule_1_operator, rule_1_input]
# example ['year', 2, 1999]

To search metadata you need to add a 4th rule "rule_X_subtype"

Currently the ui/database is the only place i've seen these listed. (I think the api will need a list/id search for meta types)

so for bpm i search in the ui/web like this

http://music.com.au/server/xml.server.php?action=advanced_search&auth=APIKEY&version=400001&rule_1=metadata&rule_1_operator=8&rule_1_input=100&rule_1_subtype=9

Operators for metadata are using the text/tag types AND numeric types in a single list as they can be ints/strings/dates. :/

rule_1_operator	Metadata
0	contains
1	does not contain
2	starts with
3	ends with
4	is
5	is not
6	sounds like (Text Only)
7	does not sound like (Text Only)
8	is greater than or equal to
9	is less than or equal to
10	is
11	is not
12	is greater than
13	is less than

So to make this searchable in python i need to extend the advanced search method to process a 4th array if metadata is the array value of rule 1

# inputs  [rule_1, rule_1_operator, rule_1_input, rule_1_subtype]
# example ['metadata', 12, 200, 9]

[Rhythm-of-Time]

Thank you for the great documentation and examples!

I am confused about the purpose of the custom metadata feature. Near the end of Ampache's debug window, I found the custom metadata I had wanted to add:

Now this looks like I have excluded them instead of adding (which I suppose does not work). The array seems to contain the exluded custom metadata. In Custom metadata - Define field list I entered "encoding", and it also appears in disabled_custom_metadata_field_input then.

But still, no matter what I add or delete here, the Song Edit Dialog and what I see after clicking on More Metadata remains the same with all wav data. Only Encoding and Audio Codec are listed here, and I cannot remove the fields by entering them in disabled_custom_metadata_field_input.

[lachlan-00]

I wonder if because wav files are not tagged they can't be added into the files?

I think we might have to add 'tagging' into the api to allow this.

[Rhythm-of-Time]

Tagging in the API would be great!

As you said, there should be a (customizable) default list of empty fields that gets initialised on import also for wav files. As these contain very few information, they otherwise will have only Encoding and Audio Codec as Meta Tags.

There is the Broadcast Wave Format, which has extended information inside the files (if that is what you mean). But it is not widely used, and there are compatibility isssues.

[lachlan-00]

okay, @DjTuxeedo and @Rhythm-of-Time this is almost done here. (at least for python)

if we look at methods to add for metadata i have at least these

Metadata Action API Methods

Update tags and metadata for files.

• [ ] metadata_fields (list the custom metadata fields)
• [ ] metadata_field (Get a single metadata field by name)
• [ ] metadata_create (create a new metadata field)
• [ ] metadata_edit (edit a metadata field)
• [ ] metadata_remove (remove a metadata field)

Then when we think about tagging i think they go into the object itself. this will edit anything in the db table for each object (e.g. name, prefix, etc) this will have to have write_id3 enabled to be allowed as the tags will overwrite when read again. (i'm not sure if this will edit vorbis/flac files?!)

• [ ] song_edit (edit a song field)
• [ ] artist_edit (edit an artist field)
• [ ] album_edit (edit an album field)

I've added the code into Ampache api for the custom fields and python to allow advanced metadata searches so if you're happy with these new methods i'll close this issue and put the new methods in the api project

https://github.com/orgs/ampache/projects/1

[DjTuxeedo]

I'm sorry @lachlan-00, It will take me a while to grasp what you have done. My programming skills are very much limited. I very much appreciate your work, its just a little above my head i guess :-) What I originally was trying to accomplish was: Using the python api - create a .csv file for every playlist in Ampache. First column is the path/filename, second column is Title, third column is the Artist and last column is the BPM value. Those .csv files then serve as an Input for an xwax instance for DJing.

My first problem seems to be that: ampache.ping(ampache_url, ampache_api, 'json') returns: false As opposed to: ampache.ping(ampache_url, ampache_api, 'xml') returns the session key

Setup is: Ampache 4.1.0 ampache.py is from January 31st from github

[Rhythm-of-Time]

I can pretty much say the same - it takes time to think and analyse... Thanks a lot for your work so far, and let's leave this open a bit.

For now, I am researching some questions: In my test database containing many wav and a few mp3, the metadata_field table looks like this:

Where do the ids and names in the table metadata_field come from? What is the "Public" column used for? It looks like in catalog.class.php the method update_song_from_tags takes the decision which metadata tags are included by default. vainfo.class.php is also doing some work cleaning up tags (metadata) which are not found in the database. I see many metadata mentioned here, but the ISRC is not mentioned.

There is no "default list" of tags. But regarding Ampache's many features there is definitely need for more tags than the standard, depending on use case. How to configure this, i.e. specify which metadata to use (display) and define data types and field lenghts? Or is this included in your project schedule already...

In general, it should be possible to write tags also in Vorbis and FLAC (as Vorbis comments only). I am not sure if it's advisable to overwrite metadata within files. Optionally they should be kept as is with additional metadata in the Ampache database.

[lachlan-00]

Where do the ids and names in the table metadata_field come from?

annoyingly they are ad-hoc based on when they are read from the tags of the song and added to the database.

What is the "Public" column used for?

Probably to allow a user to create their own custom tags but i don't think it's used anywhere.

The metadata right now i can see is only for songs and only created publicly. I think the only way it's been used previously has been in the ui.

[Rhythm-of-Time]

I stuffed an MP3 with all ID3v2.3 tags I could find. Now the metadata_field table looks like this:

INSERT INTOmetadata_fieldVALUES (1,'encoding',1),(3,'audio_codec',1),(12,'disk',1),(13,'totaldisks',1),(14,'band',1),(15,'copyright_message',1),(16,'track_number',1),(17,'conductor',1),(18,'rating',1),(19,'release_type',1),(20,'original_year',1),(21,'bpm',1),(22,'content_group_description',1),(23,'encoded_by',1),(24,'encoder_settings',1),(25,'file_owner',1),(26,'file_type',1),(27,'initial_key',1),(28,'isrc',1),(29,'length',1),(30,'lyricist',1),(31,'media_type',1),(32,'remixer',1),(33,'internet_radio_station_name',1),(34,'internet_radio_station_owner',1),(35,'original_album',1),(36,'original_artist',1),(37,'original_filename',1),(38,'original_lyricist',1),(39,'release_time',1),(40,'subtitle',1),(41,'url_artist',1),(42,'url_file',1),(43,'url_source',1),(44,'commercial_information',1),(45,'copyright',1),(46,'url_payment',1),(47,'url_publisher',1),(48,'url_station',1),(49,'album_sort_order',1),(50,'performer_sort_order',1),(51,'title_sort_order',1),(52,'album_artist_sort_order',1),(53,'composer_sort_order',1),(54,'url_user',1);

GetID3's guess looks pretty much complete. IDs 2 and 4 - 11 do not appear here, they are probably the ID3v1 stuff like album, artist, title and so on that appear elsewhere as song properties.

Now that the database has been fed with the dummy data, it would be clear which ID to adress by a newly created field (if we do not want to create an entirely new one - that would require a new entry in metadata_field). Also custom metadata could go there - if the API can change this table later.

It's a bit awkward if we pick up the way GetID3 guesses the tags on the fly, but by using it's ID scheme we at least stay compatible with the import procedure as it is now. So a workaround would be to set up the metadata fields table similar to the above.

Maybe some tags are missing, still? Barcodes are not here - should be an album property, just like catalog numbers. You adressed these already in the latest update.

[lachlan-00]

I think a good goal would be to move more file tags into song/song_data album/artist into those tables.

This would ideally reduce the need for enabling this feature.

Things like bpm and file specific tags should at least be put into song data.

If you can split up your metadata list into

Song Album Artist

I think I can just work them into those tables

[lachlan-00]

@DjTuxeedo same for you. If you can send me a list of tags or file info you want to pull out of the custom metadata I think it would be better in the object tables.

After using it I don't really like it compared to putting it in the tables

[Rhythm-of-Time]

Sorry for taking long time for thinking on this... I was digging through some implementations that I have to deal with. The largest metadata set (Soundmouse) features 176 columns. It would certainly be exaggerated to include all these in Ampache's tables - this needs to be reduced to a reasonable amount.

Though it is not very likely that we'll frequently have that 6th arranger Soundmouse provides for, sometimes a second composer / publisher may be present. There still are cases where "custom data" are necessary. One example would be entering a track ID number from a collecting society. It would probably be too much to list all of the societies in Ampache. Instead a custom field should be created. I would propose extending the existing metadata table to maintain backwards compatibility with existing collections - or is there a smarter way?

Most of the metadata should go into Song: Additional Performers / Artists BPM Label (Labelcode) ISRC GTIN (UPC) ISWC Composer Composer IPI Publisher Publisher IPI Genre Mood Keywords Instrumentation Track Description Track Internal ID

Album Album Title Album Artist Album Description Label Labelcode GTIN (UPC / EAN) Catalogue Number Release Date

Artist Artist IPI

By the way, I am just finding that two weeks ago three artists "got lost" from the database (that I am sure I did not delete). Let me see what else happened this day... did I update... The first artist I created had ID 19 last time I saw him. I wonder if recreating the records and counting up IDs does not create trouble somewhere.

But no: That day, I had tried to Disable Custom Metadata and did a Database Update. This rescanned the files (which are wavs without artist tag). It seems that the rescan killed the artist I had created and "attached" to this file. Weird.

[lachlan-00]

Did it make the wav files orphans? Or delete other data in the columns for song?

The garbage collection is based around the song table so it would clean up anything not in a file.

I don't have a lot of wav files but I'll have to put some in and try to follow the process. We need to make sure if it doesn't find tags or creates empty ones we don't remove manual ones

[lachlan-00]

Ampache has a pretty bad history for untagged file types but I think we can work around it.

[Rhythm-of-Time]

Well, it made all the wav files orphans, deleted all artists and all albums which were not from a MP3 file. Thorough job. I did not expect this - garbage collection seems to relies on what is in the files and removes the manual entries (which are not present in the wav files).

I think it would be best to have a separate metadata record for each track, in a simple table instead of the construct currently used. I mean the columns containing all metadata names / types, and the rows identified by song / track id. There should also be a column for the file path, track title, and library (possibly sublibrary) name which is the Ampache catalog(s). It would be very handy to be able export this table in a variety of ways (for fingerprinting, registration and reporting).

Additionally, we need a method for adding columns ("custom data") in case the default metadata (columns) are not enough - which we should try to avoid, and a tool to synchronize this with file metadata (if wanted / possible). The getid3 methods could be kept in order not to change too much code at a time, but they need to be tamed. Currently, we don't really have control.

Maybe you already have an idea how to deal with this? I don't really know where to start - but I am ready to help if I can.

[lachlan-00]

Verify will read for tags and replace the data. I know what to do to fix that and why it happened to you at least. Ampache replaced the info with the tags it read. (Which were empty) :/

There is already song and song_data which is meant for the extra fields. The song table itself probably doesn't need to be expanded further.

First step is stop replacing tags with empty fields. Garbage collection looks for things not attached to a file and deletes so we have to stop that.

Second step is add extra columns for the most common missing fields to at least cover all the default id3 tags.

3rd it's a good idea to have wav files ignored by the vainfo class so it doesn't accidentally overwrite itself.

I've done a bit of tag movement last year so it's not a big deal to add more. Basically I just add the columns in the db, vainfo and song/album/artist classes.

Adding custom fields is probably staying in custom_metadata. Songs are static IDs until deleted / moved so that's the best spot for edge cases.

Lastly we support multiple genres so I want song_data to store a full artist array for artist ids in each song.

Double lastly I want to separate song artist and album artist into separate search terms.

Artist Artist (song) Artist (album)

Then provide an option to list and search albums /artist pages when they are the album artist. (Very highly requested feature)

[lachlan-00]

I forgot to clarify that album and artist should be deleted if present. They might be in object_count but in user activity it's a single row