GetUploadSession error 404: the request was rejected

[ibaaj]

pinned: here's a workaround to test: https://github.com/simon-weber/gmusicapi/issues/380#issuecomment-156743098

---

Hello,

Everytime I try to upload using gmupload.py from gmusicapi-scripts to google music I get :

GetUploadSession error 404: the request was rejected

for every song.

Then if I retry I get `ALREADY EXISTS``.

I can't see those songs in google music.

gmusicapi 7.0.0 OpenSSL 1.0.1k 8 Jan 2015 Python 2.7.9

log from gmusicapi.log

2015-11-11 13:51:46,380 - gmusicapi.Musicmanager1 (shared:201) [DEBUG]: ProvideSample(args=[u'/path/to/my/song/1.mp3. ...', <gmusicapi.protocol.upload_pb2.SignedChallengeInfo object at 0x7fd2d5d059b0>, <gmusicapi.protocol.locker_pb2.Track object at 0x7fd2d5d05a28>, 'XX:XX:XX:XX:XX:XX', ''], kwargs={})
2015-11-11 13:51:46,625 - gmusicapi.Musicmanager1 (shared:242) [DEBUG]: response_type: SAMPLE_RESPONSE
sample_response {
  track_sample_response {
    client_track_id: "XXXXXXXXXXXXXXXXXXXXXX"
    response_code: UPLOAD_REQUESTED
    server_track_id: "XXXXXX-XXXXXX-XXXXXX-XXXX"
    expect_match: false
  }
}
policy {
  pause_uploads: false
  abort: false
  retry_in_minutes: 5
  bandwidth_cap_kbps: 12000
  pause_downloads: false
  download_bandwidth_cap_kbps: 16000
  send_analytics: false
  enable_gapless: true
  enable_m4p: true
}

2015-11-11 13:51:46,631 - gmusicapi.Musicmanager1 (shared:201) [DEBUG]: UpdateUploadState(args=['start', 'XX:XX:XX:XX:XX:XX'], kwargs={})
2015-11-11 13:51:46,674 - gmusicapi.Musicmanager1 (shared:242) [DEBUG]: response_type: SAMPLE_RESPONSE
sample_response {
}
policy {
  pause_uploads: false
  abort: false
  retry_in_minutes: 5
  bandwidth_cap_kbps: 12000
  pause_downloads: false
  download_bandwidth_cap_kbps: 16000
  send_analytics: false
  enable_gapless: true
  enable_m4p: true
}
2015-11-11 13:51:46,681 - gmusicapi.Musicmanager1 (shared:201) [DEBUG]: GetUploadSession(args=['XX:XX:XX:XX:XX:XX', 0, <gmusicapi.protocol.locker_pb2.Track object at 0x7fd2d5d05a28>, u'/path/to/my/song. ...', u'XXXXX-XXXXX-XXXXX-XXXXXX', False], kwargs={})
2015-11-11 13:51:46,884 - gmusicapi.Musicmanager1 (shared:242) [DEBUG]: {u'errorMessage': {u'additionalInfo': {u'uploader_service.GoogleRupioAdditionalInfo': {u'completionInfo': {u'status': u'REJECTED', u'customerSpecificInfo': {u'ResponseCode': 404}}, u'requestRejectedInfo': {u'reasonDescription': u'agent_rejected'}}}, u'reason': u'REQUEST_REJECTED', u'upload_id': u'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}}
2015-11-11 13:51:46,884 - gmusicapi.Musicmanager1 (musicmanager:538) [DEBUG]: problem getting upload session: the request was rejected
code=404 retrying=False
2015-11-11 13:51:52,888 - gmusicapi.Musicmanager1 (musicmanager:549) [WARNING]: giving up on upload session for '/path/to/my/song/1.mp3': GetUploadSession error 404: the request was rejected
2015-11-11 13:51:52,893 - gmusicapi.Musicmanager1 (shared:201) [DEBUG]: UpdateUploadState(args=['stopped', 'XX:XX:XX:XX:XX:XX'], kwargs={})
2015-11-11 13:51:52,937 - gmusicapi.Musicmanager1 (shared:242) [DEBUG]: response_type: SAMPLE_RESPONSE
sample_response {
}
policy {
  pause_uploads: false
  abort: false
  retry_in_minutes: 5
  bandwidth_cap_kbps: 12000
  pause_downloads: false
  download_bandwidth_cap_kbps: 16000
  send_analytics: false
  enable_gapless: true
  enable_m4p: true
}

[simon-weber]

Hm, my build just passed: https://travis-ci.org/simon-weber/gmusicapi/jobs/86413496.

"agent_rejected" seems to suggest something about your uploader id. What format are you using?

[dickp]

I'm seeing the same issue. I'm using the gmusicapi-script tool 'gmsync', with the default uploader-id. This has worked perfectly in the recent past.

gmusicapi v7.0.0, python 2.7.6

[ibaaj]

Hi, I use my default eth0 HWaddr (uppercase) as uploader id. I can see my device in GMusic's settings. I retried with a fake one

MacAdress=$(od -vAn -N8 -tu8 < /dev/urandom |md5sum|sed 's/^\(..\)\(..\)\(..\)\(..\)\(..\).*$/02:\1:\2:\3:\4:\5/'|tr '[:lower:]' '[:upper:]')

no change... still rejected, still "already exists"...

[simon-weber]

Weird. When I have a chance I'll look into what my build is using, but I'm pretty sure it's a mac address of that format.

[kermes-vermilio]

I get the same issue. I have been testing it with a batch of approx. 36 songs. Interestingly there are two that will pass and upload with no issues. The rest will give me the 404 error the first attempt, and will tell me they already exist thereafter.

I have attempted to use a unique ID, but that then asks me to re-confirm my account (following the URL and pasting the key) and then fails auth completely - maybe I am doing that wrong...

[0xcaff]

I'm experiencing the same issue. It seemed to work fine until around the beginning of this week. I'm also out of my of my 10 authorized devices, so I can't try with a different id.

[thebigmunch]

Just to jump on this and stay subscribed, I can confirm this behavior from both my scripts and the python interactive shell. Some songs work, some don't. The ones that don't are shown as already existing but not showing in library through gmusicapi, Google Music website, nor Android app.

[kermes-vermilio]

OK I managed to work out what I was doing wrong with the unique ID (i.e. I stopped being a bellend :-D), and that successfully logged on and gave me the "404: request rejected" (rather than the "already exists" message I have been having since the first failure.

[idvoretskyi]

Approve that bug exists - I have the same issue rigth now.

[simon-weber]

Since nobody has figured out a pattern to their failing requests, I'm guessing google's request format changed. I'll have a chance to poke at the music manager this weekend -- hopefully it's not a big change.

edit: come to think of it, has anyone successfully uploaded a non-matched track? If we're failing on the upload session, it might be matched tracks that are still succeeding.

[simon-weber]

Huh. I just captured two non-matching uploads on the newest Music Manager. Both of them got the agent_rejected response (though with a 503, not a 404), retried with a single field changed "content": "jumper-uploader-title-42" -> "content": "jumper-uploader-title-6335", and then got a successful session.

I still need to recreate the error (since it's not happening from my builds); hopefully I'll be able to test this fix without pulling in you folks.

[simon-weber]

Alright, I'm not sure if that part is relevant. I think the problem may actually be with scan-and-match. Tracks that Google doesn't request a sample of still upload fine.

This bit is new: fingerprint_algorithm: LIGHTMP3FP_CLIENT. I think this new step is causing problems both when scan-and-match is on, and when it's off (which we implement by sending an empty sample).

I'm going to see if an old music manager breaks on this step too. Hopefully it doesn't, which means there's some kind of backwards compatibility we can use. If it does, I'll have to upgrade to the new protos and go from there.

[simon-weber]

Old music managers work fine, so we don't need to worry about the new fingerprinting algorithm.

I think client ids are part of the problem, though. I've always computed a different one from google, and it would seem that the new serverside code handles this situation differently. Some experiments:

• vanilla gmusicapi upload: 404 already_matched, server id not in library
• ^, but clientid set to what MM would compute, matching enabled: successful match
• ^, but matching disabled: successful match!? gmusicapi recognizes this and logs an error "track matched without matching enabled"

So it would seem that the sever now believes the client based on client id alone? The first case is really weird: it's almost as if the server is matching you to a track in someone else's library.

Hopefully the fix is just to match MM's client id generation.

[simon-weber]

Ok, here's a workaround for you folks to test:

• download eyed3 (pip install eyed3)
• pick a song that failed to upload and make a copy of it
• run eyeD3 --remove-all copy.mp3
• upload copy.mp3 with gmusicapi

In my tests this makes the upload successful, which validates the theory about clientids.

[dickp]

The eyed3 workaround didn't work for me.

Without 'scan and match' gmsync got the error 'foo.mp3 was matched without matching enabled'

With 'scan and match', gmsync looked like it was successfully uploading but then got the error "rejected: TrackSampleResponse code 4: ALREADY_EXISTS(057b86cc-03ae-370e-aad3-4f6065ba8ac5) (1/1) Failed to upload /tmp/foo.mp3 | ALREADY EXISTS"

[simon-weber]

@dickp I think that's the expected behavior. Can you check if the first attempt resulted in a song being added to your library?

[dickp]

I can't see anything new in the 'Last added' list.

[simon-weber]

Hm. @dickp, do you mind sending me the file so I can test with it? I wonder if it has something like ape tags throwing off the hash.

[dickp]

I've emailed the file to you directly.

[simon-weber]

Weird, that uploaded fine for me once I stripped the tags.

[idvoretskyi]

@simon, BTW, is there any way run gm*.py scripts in debug mode?

[thebigmunch]

@idvoretskyi: What do you mean by debug mode? Any logging messages from them are output to stdout. I haven't yet added writing to disk, but you could simply redirect the output to a file. To enable gmusicapi's logging while using them, use the -l option as stated in the usage docs and the scripts' help.

[idvoretskyi]

@thebigmunch: I'm speaking about extended output when I run gmusicapi scripts - I receive only errors like '404', but I'd like to see the whole requesting process while running them.

[kermes-vermilio]

I tried using the eyed3 method on a batch of songs. now, from a list of 23 tracks, it recognises 3, matches 2, and attempts to upload 1 (but already exists). None of the tracks are in my library.

[simon-weber]

Huh. So what could be different about our requests that mine are working, but nobody else's are? I even had success with a track that somebody else did not. I'm on the develop version, but there aren't any changes affecting the music manager in it.

[soncaokim]

I have the issue as well (using the gmusic-script to upload songs). Worked around by changing tag data a bit (few spaces in comment) and retry again.

Nice piece of software by the way

On Tue, Nov 17, 2015, 23:37 Simon Weber notifications@github.com wrote:

Huh. So what could be different about our requests that mine are working, but nobody else's are? I even had success with a track that somebody else did not. I'm on the develop version, but there aren't any changes affecting the music manager in it.

— Reply to this email directly or view it on GitHub https://github.com/simon-weber/gmusicapi/issues/380#issuecomment-157406556 .

[simon-weber]

Maybe it's the client id generation mismatch as I described before, but our results differ because of our past uploads? It seems super unlikely that the hashes would collide that much, though.

[kermes-vermilio]

After using eyeD3 to remove the tags, I tried using id3tool to add some tags back in. This time It recognised that the song needed uploading, and tried, but got hit with the 404 again.

[simon-weber]

@idvoretskyi I've created a branch that will log entire requests: https://github.com/simon-weber/gmusicapi/tree/proto_debug. Entire responses are already logged, so this should give you everything you're interested in.

[simon-weber]

I've run a bunch of experiments and fixed a few minor differences in the scotty request, but still everything points back to the clientid.

For every track I've tried, the eyed3 workaround makes our clientid match the music manager's, and results in a successful upload with no other differences to the request. I don't have any idea why this isn't reliable for other people. I'm going to work on adding it to gmusicapi anyway -- even if it's not perfect, it's an improvement to the current situation.

[accandme]

Hello,

First of all, thank you @simon-weber for this great API!

I recently started facing the same problem, and the eyeD3 --remove-all trick did not solve the problem. So, for now, I am completely stuck, because I cannot upload any MP3.

It would be really nice if you manage to find the source of this error and fix it.

[simon-weber]

@accandme can you email me one of the original files that failed to upload? I'll try the eyed3 workaround on my end and see if I get the same result.

[dickp]

After being busy all week and not touching gmusic, I now see in 'Last added' the track I uploaded without tags. It definitely wasn't there when I looked just after attempting the upload.

So maybe it can take a while for the backend to update?

[accandme]

@simon-weber I sent you a sample song that fails.

@dickp 24 hours after attempting to upload, I still cannot see anything (new) in the gmusic interface. How long did you wait?

[simon-weber]

@accandme's track is an odd one: it's the first I've seen where eyed3 makes the clientid match Google's, but then still fails to upload. I tried it on two accounts in case unsuccessful uploads put the backend into a weird state, but it fails even I upload the tagless version first.

[simon-weber]

Oh, interesting, I think I managed to get the Music Manager into the state that @dickp saw: it's giving me ALREADY_EXISTS for a serverid that's not in my library, and the Music Manager is fine with it. Maybe the track will show up after waiting a while? It's been ~5 minutes already.

[patrickjahns]

I am experiencing the same issue - roughly 2 weeks ago it was fine with uploading tracks. Now I tried to sync my collection again with recently added tracks. First it failed with "request rejected" and when trying again I receive "Already exists" - but the tracks don`t show up in my recently added list and I can also not find them under search

This is really strange - using music manager from google the tracks upload fine (and the last change was somewhat around August for their client)

[dickp]

@accandme I don't know for sure, I just noticed it when looking this morning. It was around a week after I last looked.

[simon-weber]

@patrickjahns re "the last change was somewhat around August for their client": Google switched to a new backend mid-september and I had been assuming this behavior was related.

[ezhikus]

I see in request to https://android.clients.google.com/upsj/upauth made by Music Manager 1.0.216.5719 new parameters (there were only MAC and computer name earlier) - MAC one more time and some long id. Then this id is also used in /upsj/clientstate request body. Could it be Google added some param we don't have?

[ezhikus]

Additional information. This new id I told above is string (64 chars) and it is present in every request (and in some responses). In request to "uploadsj/scottyagent" this parameter called "UploaderId". So my guess Google decided to switch from MAC adress as uploader id to some other id - this string. The initial request to "upsj/upauth" is something like a way to map "old" uploader id (mac) to a "new" and then only new id is used in requests.

Does it make sense?

[iantpryor]

Thank you @simon-weber, I'm able to upload tracks using the eyeD3 --remove-all workaround to strip them of their tags. Songs with tags still give the ALREADY EXISTS error for me.

[simon-weber]

@ezhikus I believe that the new uploader id is a hash of the mac address. I didn't see a difference when switching from old to new in my requests, so I didn't think it was the cause of the problem (but I also didn't dig into it very much).

[jmk2000]

I'm not sure if it's relevant but I noticed that when gmusicapi uploads were complete the tracks would not show up on my sonos player. The album showed but no tracks appeared to be on it. I would have to change anything in the tags using web interface for the tracks to show on sonos.

Tracks uploaded by google music manager or via web interface showed up straight away.

[simon-weber]

@jmk2000 huh, that's bizarre. Were those tracks uploaded with the eyed3 trick? If so, maybe sonos doesn't handle missing metadata fields well.

[adam-uk]

Uploading stopped working for me, so I uninstalled what looks like quite an old version - gmusicapi-3.1.1_dev-py2.7.egg-info, and now have version (gmusicapi-7.0.0). I'm currently using https://github.com/Stono/google-music-sync as it worked OK for me.

I managed to upload 1 track. Then the next..

: GetUploadSession error 404: the request was rejected (1/1) Failed to upload 08-10-2015.mp3 | ALREADY EXISTS

and then subsequent requests

rejected: TrackSampleResponse code 4: ALREADY_EXISTS(bb9971d7-db54-3bf0-820f-3df143d57f23) (1/1) Failed to upload 08-10-2015.mp3 | ALREADY EXISTS

Hope you can resolve it :-)

[jmk2000]

@simon-weber the tracks were not changed at all; this was before the 404/ALREADY_EXISTS started.

It's making me think though that there has always been something different about the metadata of content uploaded by the API which until recently was tolerated by the backend

[antoniocosta]

Same problem here. First attempt to upload a song: GetUploadSession error 404: the request was rejected All subsequent attempts: ALREADY EXISTS Python 2.7.3, gmusicapi 7.0.0

[accandme]

Guys, do you think there is hope that this will be fixed, one day?

[simon-weber]

I think so. The eyed3 workaround (https://github.com/simon-weber/gmusicapi/issues/380#issuecomment-156743098) at least works sometimes. The next step is to do the same tag stripping when calculating the clientid.

I've been pretty busy with other things recently and haven't spent time on it since last weekend.