Different handling for same files

[ghost]

Hard to explain, hope you'll see it:

{"files": {"audio/mp3": "/track/1/convert?mimetype=audio%2Fmp3", "audio/ogg": "/track/1/convert?mimetype=audio%2Fogg"}, "album": {"id": 1, "uri": "/album/1"}, "length": 217, "number": 0, "title": "We Are The Pit Police", "slug": "we-are-the-pit-police", "artist": {"id": 1, "uri": "/artist/1"}, "bitrate": 297501, "id": 1, "uri": "/track/1"}

ARGH! Now at this moment I can't find any correct DB entry. After last index all entrys are changed:

"audio/mp3" is convert but have to be

'ftp://$$user$$:$$pass$$@URL/Folder1/Folder2/...'

Its important for me to get the complete localtion because after indexing I move the files to another location.

It worked (as I know) for some days and now not. Within the last 15 minutes I seen (some) correct entrys and now... nothing anymore.

I read the data directly from server with "/tracks"

locale.py:

# -*- coding: utf-8 -*-
from shiva.media import MediaDir, MimeType

SECRET_KEY = ''  # Set this to something secret.
MEDIA_DIRS = (
     MediaDir(root='/home/desktop/', dirs=('/Musik',), url='ftp://$$user$$:$$pass$$@URL/Folder1/Folder2/'),

)
# Add your own mimetypes here.
# MIMETYPES = (
#     MimeType(type='audio', subtype='mp3', extension='mp3',
#              acodec='libmp3lame'),
#     MimeType(type='audio', subtype='ogg', extension='ogg',
#              acodec='libvorbis'),
# )
LASTFM_API_KEY = 'xxxxx'
SCRAPERS = {
    'lyrics': (
        'azlyrics.AZLyrics',
        'metrolyrics.MetroLyrics',
        'letrascanciones.MP3Lyrics',
    ),
}
METROLYRICS_API_KEY = '1234567890123456789012345678901234567890'
BANDSINTOWN_APP_ID = 'SHIVA_APP_ID'

[tooxie]

I've encountered this error. When Shiva can't find a file in filesystem assumes that it's because it has not yet been converted. When you remove the original, all you will get are non-working /convert URIs.

What should Shiva do when the original file is not available?

[ghost]

Ah very good question... shiva should assume that I know what I'm doing.

If the original isn't available shiva should delete it or set a special flag. But to set a non-working convert URI is a bad way in my eyes.

My music lay's on "Bitcasa". The file access is very slow on this "cloud drive". That's why I index them before putting on that drive. After putting them on Bitcasa I can access them via FTP.

What to do? Think I have to change the behavior by myself, nor?

[tooxie]

I don't understand what you mean by "shiva should assume that I know what I'm doing". Let me explain why it behaves like that:

• Step 1:

You configure Shiva to scan for certain directories. Let's say you have this settings:

MEDIA_DIRS = (
    MediaDir('/home/fatmike/music/shiva'),
)
MIMETYPES = (
    MimeType(type='audio', subtype='mp3', extension='mp3',
             acodec='libmp3lame'),
    MimeType(type='audio', subtype='ogg', extension='ogg',
             acodec='libvorbis'),
)
ALLOWED_FILE_EXTENSIONS = ('mp3',)

So far, Shiva will scan /home/fatmike/music/shiva only for mp3 files and then serve them in mp3 and ogg format.

• Step 2:

You run Shiva and query for a track, let's say you get something like:

{
    "slug": "dinosaurs-will-die",
    "title": "Dinosaurs Will Die",
    "uri": "/track/510",
    "id": 510,
    "files": {
        "audio/mp3": "http://127.0.0.1:8001/nofx-pump_up_the_valuum/04. Dinosaurs Will Die.mp3",
        "audio/ogg": "/track/510/convert?mimetype=audio%2Fogg"
    }
}

• Step 3:

You request track['files']['audio/ogg'], Shiva converts it for you and gives you the file. The next time you request the same resource, you will receive:

{
    "files": {
        "audio/mp3": "http://127.0.0.1:8001/nofx-pump_up_the_valuum/04. Dinosaurs Will Die.mp3",
        "audio/ogg": "http://127.0.0.1:8001/nofx-pump_up_the_valuum/04. Dinosaurs Will Die.ogg"
    }
}

How does Shiva know if it should serve a /convert URI or a path to the file? By checking in filesystem if the file exists or not. What Shiva doesn't know is which is the original file, if the mp3 was converted from the ogg file or vice-versa. But it doesn't matter actually, because you can convert (theoretically) from any format to any other format.

And this behaviour provides the flexibility for things like scanning your collection in a format, let's say flac, and serve it in a complete different set of formats, like mp3 and ogg. In that case you would get only /convert URIs when you request a resource, and it would actually work.

In this specific case you will have to extend the MediaDir object to make it fit your expected behaviour. The urlize() method has the resposibility of receiving a path to a file and making a URI with it. Also, in the Converter class take a look at the converted_file_exists() method and all the places where it's used, you may need to adapt those as well.

When you do, please let me know, I'm really interested in seeing what you come up with, it could be nice to merge it in Shiva.

Thanks!

[dbrgn]

I think when Shiva doesn't find a file, it should set a flag, which could then be used for maintenance/cleanup scripts. It shouldn't delete entries on its own though.

There could be an option somewhere to force removal of nonexisting files, similar to rsync.

[tooxie]

I thought about flagging non-existing files but wanted to keep the track object as simple as possible. This is also quite delicate, in a local filesystem is clear, simple and fast to do the check, but what about remote filesystems? Do we want to hit a remote resource (which may include authentication also) every time a track is requested? For complete discographies that would be really unacceptable.

You can always extend the Converter and redefine what exists mean and the conversion process (which could be download the file, convert it, and upload it back). Also by extending MediaDir you can redefine the URIs for accessing (and converting) files. So this is currently possible.

@dbrgn What do you mean by removing nonexisting files?

[ghost]

---Also by extending MediaDir you can redefine the URIs for accessing (and converting) files. So this is currently possible.

In my case this isn't possible. I didn't get the complete URI 'cause I moved the File to my FTP server after indexing. So I only get the "convert URI". Why shiva gives the convert URI if the file isn't present?

That's the reason I opened this issue.

MP3 is stored in:

/home/desktop/Musik Then I index the files and after that I move them to my FTP. That's why I provide the url='ftp://$$user$$:$$pass$$@URL/Folder1/Folder2/'. But after next index process Shiva do not find the file and give a convert URI. But what should shiva convert if the file isn't found?

Only two possible solutions:

Print out the URI normal how I want: ftp://$$user$$:$$pass$$@URL/Folder1/Folder2/xxxxx.mp3 Give nothing

[tooxie]

In my case this isn't possible. I didn't get the complete URI 'cause I moved the File to my FTP server after indexing. So I only get the "convert URI".

As I said, extending the MediaDir and Converter classes you can make Shiva return the URI that you want. But you will have to code it, there's nothing like that implemented (yet).

Why shiva gives the convert URI if the file isn't present?

I just explained that in a very long comment.

Only two possible solutions:

Print out the URI normal how I want: ftp://$$user$$:$$pass$$@URL/Folder1/Folder2/xxxxx.mp3 Give nothing

Feel free to implement it.

[ghost]

Yes you've explained that. But if the file won't be found, shiva shouldn't give any URL/URI. Sure

"because you can convert (theoretically) from any format to any other format."

But if nothing can be found? :-)

Okay seems to be a very good moment to begin with python and I will do that. It will takes some days for me to extending the classes. After that I'll starts a PR for my changes. I think about a option in local.py to modify "not found files" handling.

[tooxie]

I hope you found a workaround for this. If you find any further problems feel free to open a new ticket. I'm closing this one.