Now getting a 'Failed to fetch mime-type' error message after updating

[chrism]

I’m using a pretty simple script to request the URL of an mp3 file stored online. Prior to updating liquid soap this worked perfectly and the mp3 file played (using liquid soap 1.3.1).

After switching to installing with opam and liquid soap 1.3.3 instead of the track playing it logs out an error like this:

18:39:10 soap.1 | 2019/02/10 18:39:10 [procol.external:3] Failed to fetch mime-type for http://mysite.com/krrBrS6LQ1poMoM9EyI2zhePCUk6jvMz7IkNjxLT27-ScLKeCP1TuyJcbRQ287Or6s4i22farmIePVxd-K-HwtVrwecha4qOI5CiobC-ZMgMcUfGXac_VqGgLO-7qx5H4-pm4p40CGGHhexL90uCe02JsQjJ-sJ5C0rj0cLCMIHcGQdVE6tjpUUvzGB36KMK2-VoHlJGSW0&.

The stream gets generated like this:

output.icecast(
  %mp3(id3v2=true,bitrate=128,samplerate=44100),
  s
)

• OS: Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-164-generic x86_64)
• Version: 1.3.3

Installed using opam now:

opam install cry alsa pulseaudio mad taglib lame ogg vorbis opus voaacenc fdkaac faad flac ladspa soundtouch samplerate xmlplaylist dtools duppy mm liquidsoap

Has something changed between versions which may have caused this?

I’ve looked to see if I can manually specifiy the mime-type but been unable to find anything in the documentation.

Many thanks for any advice.

[toots]

Hey,

The way to handle these has changed recently indeed. Could you send me a test url to check? The one in your logs returns a 404 right now.

[chrism]

Hi,

I’m really sorry but I changed the URL on purpose, it is an encrypted 3rd party URL which I can’t share unfortunately.

Is there some other way I can send the file to you—perhaps upload it to another URL?

I’m not sure what you’d like to test but would be happy to try to replicate the circumstances as best as I can.

Thanks.

[toots]

Sure. I you can give me the logs of this command:

curl -L -v -o /tmp/bla.mp3 <url>

That would help a lot.

The new code relies on the Content-Type HTTP header to detect the audio format and it seems that your web server does not return one.

[chrism]

I’ve tried my best to provide everything I can while keeping it anonymised.

Hope this is good enough for you?

Thanks again. for taking a look.

vagrant@dev:~/project_path$ curl -L -v -o ./tmp/bla.mp3 <url>
[1] 2371
-bash: .: filename argument required
.: usage: . filename [arguments]
vagrant@dev:~/project_path$ * Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying X.X.X.X...
* Connected to subdomain.domain.com (X.X.X.X) port 80 (#0)
> GET <url> HTTP/1.1
> User-Agent: curl/7.35.0
> Host: subdomain.domain.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: public
< Content-Type: audio/mpeg
< Date: Mon, 11 Feb 2019 21:18:11 GMT
< Expires: Thu, 11 Jul 2019 21:18:11 GMT
< Last-Modified: Thu, 07 Feb 2019 13:23:44 GMT
< Pragma: 
* Server ECAcc (pox/A551) is not blacklisted
< Server: ECAcc (pox/A551)
< X-Cache: HIT
< X-Host: blm-prxmob-13
< Content-Length: 3612838
< 
{ [data not shown]
100 3528k  100 3528k    0     0  8648k      0 --:--:-- --:--:-- --:--:-- 8690k
* Connection #0 to host subdomain.domain.com left intact

The audio file downloads successfully to the tmp directory.

[toots]

Hmm. That's interesting. It looks like the webserver does indeed pass a Content-Type header. Any way you could share that url in private so I can test? Feel free to hit me up at toots@rastageeks.org

[chrism]

Hi @toots I’m really sorry but I’ve checked with the 3rd party and they are not OK with me sharing it, even privately.

I wish I could be more help, is there a way to explicitly set the content-type or similar workaround for a situation like this? I know that the files will only ever be mp3.

Sorry again and thanks for your help.

[hubb]

pretty simple script to request the URL of an mp3 file stored online

Does this mean you're using the request.dynamic operator in your liquidsoap script?

You could try to configure the decoder's accepted mime types (here for mad)

set('decoder.mime_types.mad', ['audio/mpeg', 'audio/mpeg3', 'audio/mp3'])

Alternatively, if you use the playlist operator, you could configure the mime_type parameter

[chrism]

Hi @hubb that is exactly right, I’m using request.dynamic with a custom function.

The code looks like this:

def get_request() =
  # Get the URI
  uri = list.hd(default="",get_process_lines("curl http://website.com/api/endpoint"))
  # Create a request
  request.create(uri)
end

s = request.dynamic(id="s",timeout=60.0,get_request)

I’m going to try to understand better how to implement what you mean for my particular case.

Thanks for your help.

[hubb]

set allows you to change some Liquidsoap settings.

With set('decoder.mime_types.mad', ['audio/mpeg', 'audio/mpeg3', 'audio/mp3']), we're telling mad (the decoder) which mime-types to use to guess the audio format. You can find more information here, at "Decoder settings" > "Mime type used for guessing audio format".

Maybe one of those API call is returning a Content-Type that isn't handled by any decoder

[chrism]

Thanks, I’ve just added the line

set('decoder.mime_types.mad', ['audio/mpeg', 'audio/mpeg3', 'audio/mp3'])

to set the decoder mime types for mad but I’m still getting the same Failed to fetch mime-type error message.

[toots]

Hey.

This should work as intended if you have the correct mime-type setup.

As a side-note, since I believe 1.3.3, you can also use protocols to achieve the same result. I still need to write proper documentation for them but you can read the script files for them here: https://github.com/savonet/liquidsoap/blob/master/scripts/protocols.liq

If you cannot share the url, could you post some logs with log.level set to 4? Also, maybe the output of:

curl -sLI -X HEAD <http url>

Thanks for your patience!

[chrism]

Hi again, no problem about my patience, I really appreciate your help with this.

Here are the logs after log.level set to 4.

18:20:46 soap.1 | 2019/02/13 18:20:46 [procol.external:4] Running curl -sLI -X HEAD "http://website.com/blah" | grep -i '^content-type' | tail -n 1 | cut -d':' -f 2 | cut -d';' -f 1
18:20:46 soap.1 | 2019/02/13 18:20:46 [lang.run_process:4] Starting process
18:20:46 soap.1 | 2019/02/13 18:20:46 [lang.run_process:4] Closing process's stdin
18:20:56 soap.1 | 2019/02/13 18:20:56 [procol.external:3] Failed to fetch mime-type for http://website.com/blah
18:20:56 soap.1 | 2019/02/13 18:20:56 [lang.run_process:4] Cleaning up process

And from running the curl -sLI -X HEAD <http url> here is the result

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: public
Content-Type: audio/mpeg
Date: Wed, 13 Feb 2019 18:23:28 GMT
Expires: Sat, 13 Jul 2019 18:23:28 GMT
Last-Modified: Thu, 07 Feb 2019 13:23:44 GMT
Pragma: 
Server: ECAcc (pox/A551)
X-Cache: HIT
X-Host: blm-prxmob-13
Content-Length: 3612838

Not sure there is anything new or useful there for you, though.

I’ve taken a quick look at the script example but my initial reaction is that is a little out of my level of expertise—but I will try to work through it to see if I can make some sense out of it.

Thanks once more!

[toots]

Hey. Thanks for these info. I'm still not sure what's going on. Could you add the following to your script, this should replace the resolvers with one that has more info logged out:

# Resolve download protocols using curl
# @flag hidden
def download_protocol(proto,~rlog,~maxtime,arg) =
  curl = get(default="curl","protocol.external.curl")
  uri = "#{proto}:#{arg}"

  log = log(label="procol.external")

  def log(~level,s) =
    rlog(s)
    log(level=level,s)
  end

  env_vars = get(default=[],"protocol.process.env")
  env = environment()
  def get_env(k) =
    (k,env[k])
  end
  env = list.map(get_env,env_vars)
  inherit_env = get(default=true,"protocol.process.inherit_env")

  timeout = maxtime - gettimeofday()

  # First define using curl.
  def get_mime() =
    cmd = "#{curl} -sLI -X HEAD #{quote(uri)} | grep -i '^content-type' | tail -n 1 | cut -d':' -f 2 | cut -d';' -f 1"
    log(level=4,"Running #{cmd}")
    x = run_process(timeout=timeout,env=env,inherit_env=inherit_env,cmd)
    print("from curl: #{x}")
    if fst(snd(x)) != "exit" or snd(snd(x)) != "0" then
      log(level=3,"Failed to fetch mime-type for #{uri}.")
      ""
    else
      lines = string.split(separator="\\n",fst(fst(x)))
      string.case(lower=true,string.trim(list.hd(default="",lines)))
    end
  end

  def head_mime(~name, ret) =
    def get_mime() =
      print("From internal: #{ret}")
      status = fst(fst(ret))
      headers = snd(fst(ret))
      code = snd(fst(status))
      if 200 <= code and code < 300 then
        headers["content-type"]
      else
        log(level=3,"Failed to fetch mime-type for #{uri}.")
        ""
      end
    end
    get_mime
  end

  sub = string.sub(uri,start=0,length=5)

%ifdef https.head
  get_mime =
    if sub == "https" then
      log(level=4,"Fetching https head for #{uri}")
      head_mime(name="https",https.head(timeout=timeout,uri))
    else
      get_mime
    end
%endif

  get_mime =
    if sub != "https" then
      log(level=4,"Fetching http head for #{uri}")
      head_mime(name="http",http.head(timeout=timeout,uri))
    else
      get_mime
    end

  mime = get_mime()

  extname =
    if list.mem(mime, ["audio/mpeg", "audio/mp3"]) then
      "mp3"
    elsif list.mem(mime,["application/ogg", "application/x-ogg",
                         "audio/x-ogg", "audio/ogg", "video/ogg"]) then
      "ogg"
    elsif mime == "audio/x-flac" then
      "flac"
    elsif list.mem(mime,["audio/mp4", "application/mp4"]) then
      "mp4"
    elsif list.mem(mime,["audio/vnd.wave", "audio/wav",
                         "audio/wave", "audio/x-wav"]) then
      "wav"
    else
      log(level=3,"No known file extension for mime: #{mime}")
      "osb"
    end
  [process_uri(extname=extname,"#{curl} -sL #{quote(uri)} -o $(output)")]
end

# Register download protocol
# @flag hidden
def add_download_protocol(proto) =
  add_protocol(syntax="#{proto}://...",doc="Download files using curl",proto,download_protocol(proto))
end
if get(default=true,"protocol.external") then
  list.iter(add_download_protocol,get(default=["http","https","ftp"],"protocol.external.protocols"))
end

[chrism]

Hi again @toots sorry it took me a couple of days to get back to you on this, but I’ve just had chance to add the code. Here is the output (again, I’ve had to snip the actual URL):

10:00:17 soap.1 | 2019/02/21 10:00:17 [procol.external:4] Fetching http head for <url>
10:00:17 soap.1 | From internal: (((("HTTP/1.0",200),"OK"),[("accept-ranges","bytes"), ("cache-control","public"), ("content-type","audio/mpeg"), ("date","Thu, 21 Feb 2019 10:00:17 GMT"), ("expires","Sun, 21 Jul 2019 10:00:17 +0000"), ("pragma",""), ("server","Apache"), ("x-host","blm-prxmob-45"), ("content-length","3612838"), ("connection","close")]),"")
10:00:17 soap.1 | 2019/02/21 10:00:17 [protocol.process:4] Processing mp3,curl -sL "http$(colon)//<url>" -o $(output)
10:00:17 soap.1 | 2019/02/21 10:00:17 [protocol.process:4] Executing curl -sL "<url>" -o "/tmp/liq-process64ded5.mp3"
10:00:17 soap.1 | 2019/02/21 10:00:17 [lang.run_process:4] Starting process
10:00:17 soap.1 | 2019/02/21 10:00:17 [lang.run_process:4] Closing process's stdin
10:00:27 soap.1 | 2019/02/21 10:00:27 [protocol.process:3] Failed to execute curl -sL "<url>" -o "/tmp/liq-process64ded5.mp3": ("timeout","9.19564294815")
10:00:27 soap.1 | 2019/02/21 10:00:27 [request:4] Failed to resolve "process:mp3,curl -sL \"http$(colon)//<url>\" -o $(output)"! For more info, see server command 'trace 12'.
10:00:27 soap.1 | 2019/02/21 10:00:27 [lang.run_process:4] Cleaning up process

Let me know if there is anything else I can do to help you understand the issue.

Thanks.

[toots]

Yes, most definitely. Looks like the log message was confusing.. Seems like a simple download timeout. What source operator are you using? It should have a timeout parameter, which you should increase and see if that helps.

[chrism]

Thanks @toots but unfortunately I don’t think this is the issue because I only changed it to 10 seconds recently (during testing in this thread) from an initial 60 seconds.

This is the code I am using:

s = request.dynamic(id="s",timeout=10.0,get_request)

Before it was this

s = request.dynamic(id="s",timeout=60.0,get_request)

I can change it back if you want, but don’t think it makes a difference.

[nightwatcher74]

Hi,

I have the same problem... :( and I think i found the problem... the string of the curl request to get the mimetype has a length of 13 bytes... it looks like it is " audio/mepeg " ant this is not "audio/mpeg" could thisbe the problem??

Frank

Edit: The exact string I got back from curl has space in the beginning and a Dos/CR: " audio/mpeg\r\n" as I understand it correctly you only remove the \n and the trim the variable. does the trim remove a \r ??

[nightwatcher74]

Hi, forget my last message... it must be a deeper problem...

When i use your debug script, i have the problem that the files will be downloaded fast (I see it in /tmp) but I got an error message "Failed to execute curl..." with a timeout from about 20sec.

I got the exactly same 20sec waiting time when the script wants to get the content-type via HEAD request. with the message "unable to fetch mime-type"

The curl statements are OK and all is very fast, and the script will execute it, but has a problem to get the results correctly/waiting for end of the curl request but it's already completed!

Frank

[chrism]

hi @nightwatcher74 and @toots just to say I still have never found a solution to this and would be very keen to help resolve it in any way I can.

Thanks.

[nightwatcher74]

Hi @chrism it looks like that is is the same issue as #691 at the moment i installed a debian buster (because of the ocaml requirements) and will compile it from git...

[nightwatcher74]

Hi @chrism, @toots,

This problem has definitly it's root cause at the same at #691 & #703

I made a docker container with the following update after an opam installation:

git clone --recursive https://github.com/savonet/liquidsoap.git cd liquidsoap git checkout 1.3.7 opam pin add .

and it worked! :) Thanks to @toots

I think it will be the best to link this ticket to the other ones (and close it?!)...

Frank

[chrism]

Hi @nightwatcher74 — great news, glad you got it working with this fix!

Unfortunately, my understanding of devops is really limited (I'm design/front end) so hope you don't mind me asking but I'd like to test this.

At the moment I'm using a puppet script to build the environment and install liquidsoap using opam, as is now recommended, like this

#!/bin/bash

source /home/deployer/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

cd /home/deployer
opam install cry alsa pulseaudio mad taglib lame ogg vorbis opus voaacenc fdkaac faad flac ladspa soundtouch samplerate xmlplaylist dtools duppy mm liquidsoap

but before that I have used an alternative way, from the github repo, which looked like this

#!/bin/bash

source /home/deployer/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

cd /tmp
git clone https://github.com/savonet/liquidsoap-full.git liquidsoap
cd liquidsoap
cp ../PACKAGES .
make init
./bootstrap
./configure --enable-debugging --disable-graphics
make
sudo make install

Do you have a suggestion of how I could do a similar test as you've done with my setup?

Many thanks

Chris

[toots]

@nightwatcher74 very happy to read that. I plan on releasing 1.3.7 very soon.

[toots]

Hi @chrism. You should be able to use opam's local pinning to test specific git branches. Something like this:

#!/bin/bash

source /home/deployer/.opam/opam-init/init.sh > /dev/null 2> /dev/null || true

cd /home/deployer
opam install cry alsa pulseaudio mad taglib lame ogg vorbis opus voaacenc fdkaac faad flac ladspa soundtouch samplerate xmlplaylist dtools duppy mm 

cd /tmp
git clone --recursive https://github.com/savonet/liquidsoap.git liquidsoap
cd liquidsoap
git checkout 1.3.7
opam pin add .

[toots]

Fixed in 1.3.7. Please reopen otherwise.

[chrism]

Hi @toots sorry it's taken me so long to get back to you on this I had to put it on hold for a bit.

I tried to follow your advice above but when installing I get this error message during installation.

[ERROR] Opam has not been initialised, please run opam init

If you have any suggestions much appreciated.

==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Done.
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Cloning into 'liquidsoap'...
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Submodule 'm4' (git://github.com/savonet/m4.git) registered for path 'm4'
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Cloning into 'm4'...
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Submodule path 'm4': checked out 'a8bf3e1c8ff0662c03c5d8571357bc2b215156dc'
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: Note: checking out '1.3.7'.
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: 
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: You are in 'detached HEAD' state. You can look around, make experimental
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: changes and commit them, and you can discard any commits you make in this
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: state without impacting any branches by performing another checkout.
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: 
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: If you want to create a new branch to retain commits you create, you may
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: do so (now or later) by using -b with the checkout command again. Example:
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: 
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns:   git checkout -b new_branch_name
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: 
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: HEAD is now at 1a76f53... Set the correct m4.
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: M  m4
==> dev: Notice: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: [ERROR] Opam has not been initialised, please run `opam init'
==> dev: Error: /tmp/build-liquidsoap.sh returned 50 instead of one of [0]
==> dev: Error: /Stage[main]/Liquidsoap/Exec[/tmp/build-liquidsoap.sh]/returns: change from notrun to 0 failed: /tmp/build-liquidsoap.sh returned 50 instead of one of [0]
==> dev: Notice: /Stage[main]/Liquidsoap/File[/usr/local/bin/liquidsoap]: Dependency Exec[/tmp/build-liquidsoap.sh] has failures: true
==> dev: Warning: /Stage[main]/Liquidsoap/File[/usr/local/bin/liquidsoap]: Skipping because of failed dependencies

[chriswnl]

It's been a while but I came across this same issue with a link to a dropbox file ('private' url) in version 1.4.0. I tracked it down after some time to an issue with the get_mime() function.

It seems the new function using https.head, doesn't return the content_type due to a redirect: the call to dropbox then redirects to a dropboxcontent url.

The originally defined get_mime() function works.

%ifdef https.head
  get_mime =
    #if sub == "https" then
    #  log(level=4,"Fetching https head for #{uri}")
    #  head_mime(name="https",https.head(timeout=timeout,uri))
    #else
      get_mime
    #end
%endif

seems to fix the problem for https. I suspect the same may be true of non SSL/TLS connections too.

Hope this helps some one.

[chrism]

Thanks @chriswnl — unfortunately I had to put this on hold some time ago but you have inspired me to have another go!

[toots]

Hi all. This should be fixed in the upcoming 1.4.2 release.

[chrism]

Oh wow this is brilliant news ! :)

Thanks so much for the update, is there any idea of timeframe for 1.4.2 or should I just keep checking?

Thanks again :)

[toots]

Within a week or two.. :-)