search

mapillary / mapillary_tools

Command line tools for processing and uploading Mapillary imagery
BSD 2-Clause "Simplified" License
264 stars 137 forks source link

HTTP Error 403 Forbidden #236

Closed jaku103 closed 6 years ago

jaku103 commented 6 years ago

Basic information

Steps to reproduce behavior

I'm using the new scripts. I am receiving "HTTP error: HTTP Error 403"on every image. This is hapening after struggling with an user authentication error. That was caused from never setting a password. I always used a linked account. This is the script output:

Running user_process for 11144 images, skipping 0 images. Running import_meta_data_process for 11144 images, skipping 0 images. Running geotag_process for 11144 images, skipping 0 images. Running sequence_process for 9999 images, skipping 0 images. Running sequence_process for 1145 images, skipping 0 images. Running upload_params_process for 11144 images, skipping 0 images. Running mapillary_image_description for 11144 images, skipping 0 images. Uploading 11144 images with valid mapillary tags (Skipping 0) Uploading with 4 threads HTTP error: HTTP Error 403: Forbidden on Y0010004.JPG--------] 0.0% ... 11140 images left. HTTP error: HTTP Error 403: Forbidden on Y0010003.JPG HTTP error: HTTP Error 403: Forbidden on Y0010001.JPG HTTP error: HTTP Error 403: Forbidden on Y0010002.JPG HTTP error: HTTP Error 403: Forbidden on Y0010003.JPG HTTP error: HTTP Error 403: Forbidden on Y0010002.JPG HTTP error: HTTP Error 403: Forbidden on Y0010004.JPG

ToeBee commented 6 years ago

I was getting 403s the other day too but it works for me now.

nicola-d commented 6 years ago

Same problem on Raspbian

jernejaMislej commented 6 years ago

Hi, sorry for getting back late. We had some issues last week due to security updates, which resulted in this error. This issue is now resolved. If you continue to get this error and suspect it is because you are not able to authenticate successfully, you can run the process with --verbose and monitor if you get any errors or warnings when running user process and let me know. But you will need to enter a valid user email and password for the user name specified when running the tool, if you dont have them or have forgotten them then you should contact our customer support

nicola-d commented 6 years ago

I still have problems on Raspbian. --verbose doesn't say much more:

HTTP error: HTTP Error 403: Forbidden on VIRB0016-327.JPG----] 0.1% ... 1867 images left.

tastrax commented 6 years ago

Likewise - been battling this error for a couple of days now - https://forum.mapillary.com/t/running-the-python-scripts/130/99

After some investigation the only thing I could find was that my signature hash in uploader.py did not match that which is shown on my mapillary settings. Changed that but it made no difference.

So frustrating…!!

tastrax commented 6 years ago

image

tastrax commented 6 years ago

image

cliffordsnow commented 6 years ago

I'm getting the same error tonight.

jernejaMislej commented 6 years ago

Hi, thanks for getting back . I will look into it. @tastrax the hashes that are hardcoded in the uploader.py should NOT be the same as the ones in your mapillary settings. The hashes that are hardcoded in uploader.py are for direct uploads only, without processing. If your import requires processing, your user hashes are used. Your user hashes are obtained through authentication when using the tools for the first time. After authentication they are stored in a global config file for future usage. What you can do is check the content of your global config file, it is located in your home, in a hidden directory called .config, inside there is directory mapillary with a file config. The hashes there need to match the ones in your mapillary settings.

tastrax commented 6 years ago

Thanks for that – I have reinstalled the Mapillary tools, checked my config settings and they match the Mapillary site

Same result

Any more clues on how to get things working?

I am on Windows 64bit Win 10 Pro

Cheers - Phil

From: jernejaMislej [mailto:notifications@github.com] Sent: Friday, 8 June 2018 4:50 PM To: mapillary/mapillary_tools Cc: Phil Wyatt; Mention Subject: Re: [mapillary/mapillary_tools] HTTP Error 403 Forbidden (#236)

Hi, thanks for getting back . I will look into it. @tastrax https://github.com/tastrax the hashes that are hardcoded in the uploader.py should NOT be the same as the ones in your mapillary settings. The hashes that are hardcoded in uploader.py are for direct uploads only, without processing. If your import requires processing, your user hashes are used. Your user hashes are obtained through authentication when using the tools for the first time. After authentication they are stored in a global config file for future usage. What you can do is check the content of your global config file, it is located in your home, in a hidden directory called .config, inside there is directory mapillary with a file config. The hashes there need to match the ones in your mapillary settings.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mapillary/mapillary_tools/issues/236#issuecomment-395665761 , or mute the thread https://github.com/notifications/unsubscribe-auth/AIXm41ZvQjZi-X85KS8JozkYYrUGTXx-ks5t6h58gaJpZM4UX8ha . https://github.com/notifications/beacon/AIXm47aJPus_ISDp5WpV0_KFMEFMGiiCks5t6h58gaJpZM4UX8ha.gif

tastrax commented 6 years ago

No luck yet - even with a straight upload only

image

nicola-d commented 6 years ago

Same on Raspbian. And not much debug information with --verbose enabled.

Could it be related with urllib version?

jernejaMislej commented 6 years ago

Hi thank you for the patience. We are looking into the issue now and it looks like the issues we had last week due to security updates are still not resolved. Hopefully we manage to fix it as soon as possible, I will keep you updated.

nicola-d commented 6 years ago

Looks like it is fixed. Now works! Thank you

Alessandro-Barbieri commented 6 years ago

Now I get HTTP Error 400: Bad request on

tastrax commented 6 years ago

Still 403 for me

tastrax commented 6 years ago

Downloaded latest scripts and BINGO - full upload!

image

nicola-d commented 6 years ago

Worked for a while and now "HTTP error: HTTP Error 403: Forbidden on " again...

jernejaMislej commented 6 years ago

Hi sorry for the issues, please install the tools again and update config by either deleting the config file(~/.config/mapillary/config) and you will be prompted to enter you credentials again when running the tools or download the edit_config.py script at https://github.com/mapillary/mapillary_tools/blob/master/bin/edit_config.py and run it with python edit_config.py , you will be asked which section you want to edit, enter you user_name and then enter your credentials same as when running the tools for the first time

jrcarlsen commented 6 years ago

I am still getting HTTP 403 errors. My username have a space in it which means I have to type it in as "johnny%20carlsen" for it to work. I think the problem I am seeing could be related to this.

jernejaMislej commented 6 years ago

@jrcarlsen which os are you on? maybe better to specify your username as --user_name "johnny carlsen" quotation marks should take care of the space between

nicola-d commented 6 years ago

Installed the tools again, deleted config file and re-entered the credentials again but I'm still getting HTTP 403 errors.

My username is a single word, no spaces.

I'm on linux, Raspbian to be precise.

jernejaMislej commented 6 years ago

hi, did you install with --upgrade? ie sudo pip install --upgrade git+https://github.com/mapillary/mapillary_tools

nicola-d commented 6 years ago

no, i uninstalled everything first with pip uninstall piexif Pillow gpxpy mapillary-tools pytest ExifRead then i reinstalled as specified in the README.

The only difference is that I'm using the --user switch to install the script in my home and not globally.

Saturday I was able to upload a thousand of images or so and then nothing more because of the HTTP 403 error

jernejaMislej commented 6 years ago

ok thanks @ndriutti are you using upload or process_and_upload? if you are using process_and_upload can you check in your latest upload attempt that doesnt work, whether you have aws_key in the .mapillary/logs/*image-folder*/upload_params_process.json of the import?

nicola-d commented 6 years ago

@jernejaMislej I'm using upload. Should I give process_and_upload a try?

jernejaMislej commented 6 years ago

no @ndriutti you shouldnt have, if your images were taken with the app they can be uploaded directly with upload only and it should work but if you have previously processed the images and are now just running upload that is ok also and will usually work, but in current case you should reprocess the images to get the right upload params(rerun process with --rerun), since you likely have the previous params in .mapillary from when there were issues thank you so much for the patience and feedback

nicola-d commented 6 years ago

@jernejaMislej I reprocessed the images (taken with an action camera) with the new installation of the script and the issue seems disappeared. Thank you

jernejaMislej commented 6 years ago

great @ndriutti :confetti_ball: thank you so much for the persistence

nicola-d commented 6 years ago

@jernejaMislej there's still something strange happening. I uploaded about 150 images without getting errors but the pending images counter in my status page is not increasing...

ToeBee commented 6 years ago

That is a completely separate issue. Basically all of the "in process" information is garbage :)

Uploading images puts them into some magical buffer in the sky. Once they come out of that buffer and into the processing queue they show up as "pending" but at that point it usually only takes a short amount of time to move through the queue anyway. Then you get notified that your images have been "published" but they usually aren't actually displayed on the map yet, sometimes for a day or more. So yeah... annoying but not related to this issue.

I just uploaded 2,800 images this morning without any 403s. So yay!

jrcarlsen commented 6 years ago

@jernejaMislej I initially tried with the username being "johnny carlsen", but it failed to authenticate, @eneerhut suggested that I tried "johnny%20carlsen" instead, which succeeded in the authentication but failed with the uploads.

My user has now been renamed to "jrcarlsen" and I no longer have any issues. Photos are being uploaded successfully right now.

nicola-d commented 6 years ago

@ToeBee the magical buffer in the sky is more like a black hole. It "ate" at least a couple of thousand images but there is no trace of them... No pending, nothing published so far...

Is there anyone who can address me to the right issue (if it is already open)?

Thank you

jernejaMislej commented 6 years ago

@ndriutti what is your username, i will check for your images

nicola-d commented 6 years ago

@jernejaMislej my username is **

jernejaMislej commented 6 years ago

@ndriutti there are images uploaded in your account. 1681 images, 11 sequences, from 11,12 June, but are missing the DONE file, which is uploaded automatically unless you specify --manual_done, which results in a prompt whether you want to finalize the upload. if you specified --manual_done and did not approve finalize, that could be a reason why the images did not go through. I will create the DONE file for you for those images, so they go in the db asp.

nicola-d commented 6 years ago

@jernejaMislej Strange, I never used the --manual_done switch. Ok, so next time this behaviour happens should I open a separate issue?

jernejaMislej commented 6 years ago

@ndriutti yes please do

Alessandro-Barbieri commented 6 years ago

From my last run

Running import_meta_data_process for 153 images, skipping 0 images.
Running geotag_process for 153 images, skipping 0 images.
Running sequence_process for 153 images, skipping 0 images.
Running upload_params_process for 153 images, skipping 0 images.
Running mapillary_image_description for 153 images, skipping 0 images.
Uploading 153 images with valid mapillary tags (Skipping 0)
Uploading with 4 threads
HTTP error: HTTP Error 400: Bad Request on tt_img_00007.jpg--] 5.2% ... 145 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00007.jpg--] 8.5% ... 140 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00022.jpg--] 18.3% ... 125 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00033.jpg--] 26.1% ... 113 images left.
URL error: <urlopen error [Errno 32] Broken pipe> on tt_img_00041.jpg... 109 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00052.jpg--] 37.3% ... 96 images left..
HTTP error: HTTP Error 400: Bad Request on tt_img_00075.jpg--] 50.3% ... 76 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00074.jpg--] 51.0% ... 75 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00083.jpg--] 56.9% ... 66 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00086.jpg--] 57.5% ... 65 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00089.jpg--] 60.8% ... 60 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00099.jpg--] 68.0% ... 49 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00106.jpg--] 69.9% ... 46 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00106.jpg--] 73.2% ... 41 images left.
URL error: <urlopen error [Errno 104] Connection reset by peer> on tt_img_00116.jpgleft.
HTTP error: HTTP Error 400: Bad Request on tt_img_00119.jpg--] 78.4% ... 33 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00116.jpg--] 81.0% ... 29 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00126.jpg--] 85.6% ... 22 images left.
HTTP error: HTTP Error 400: Bad Request on tt_img_00148.jpg==] 99.3% ... 1 images left..
HTTP error: HTTP Error 400: Bad Request on tt_img_00150.jpg
Done uploading 153 images.===================================] 100.0% ... 0 images left.
tastrax commented 6 years ago

FYI - process_and_upload 2000 images today without issue - woo hoo! Only suggestion is that it would be good to get a message (sending done file..?) at the end of the upload as it hangs on the message below for a long time before returning the focus to python

Done uploading 2007 images.=============================] 100.0% ... 0 images left...

Otherwise all good.

Cheers - Phil

jernejaMislej commented 6 years ago

@tastrax great work :confetti_ball: @Alessandro-Barbieri unfortunately these result from network issues, so there is not much I can do to help. It happens every now and then and images always end up uploaded.

ToeBee commented 6 years ago

403 errors are back. I JUST uploaded a short sequence successfully and tried to upload the next directory and splat

tastrax commented 6 years ago

Is this with the latest scripts Toby @ToeBee ? There was an update a few days ago. I am heading out shortly to collect some images so I will also try and upload later in the day and see if I get the same.

ToeBee commented 6 years ago

It seems it was a temporary glitch. I tried again and the 305 image sequence just finished successfully. And yes, I did update the script earlier tonight to the 0.0.2 version.

jernejaMislej commented 6 years ago

hi please install the latest release v0.1.0 and open an issue if any problems