search

OneDrive / onedrive-api-docs

Official documentation for the OneDrive API
MIT License
457 stars 231 forks source link

Forbidden response when downloading chunks #178

Closed saguiitay closed 8 years ago

saguiitay commented 9 years ago

I'm trying to download a file in chunks. I'm making the following request:

GET https://api.onedrive.com/v1.0/drive/items/82A4826DAA392C28!5146/content
Headers:
Authorization: bearer EwCA...ZxAQ==
Range: bytes=0-2097152

And I'm getting a valid 302 Found response:

StatusCode: 302
Headers:
X-WLSPROXY: BN1302____PAP237
X-MSNSERVER: BL3301____PAP110
X-AsmVersion: UNKNOWN; 19.22.0.0
X-AsmVersion-ProxyApp: UNKNOWN; 19.22.0.0
Date: Wed, 05 Aug 2015 13:31:52 GMT
Location: https://public.bl3301.livefilestore.com/y3m9aaOZd0wyyDVSJDL_DI_v0K1mU8NwXMjZ37sB6bOF7_Vq2L4AwcfuqFgO3IeJdZkp_U1q9Dq7O26O0lXJrExCDzXcv1kmex-tapAWQ8M6v298Mb7_jVzda2YBZnTTwtlCHVSY95mM6h1W7mVWaESLczOzfUTyHbP3dui4OSu8f0co8M_4TyE4w6poveTN3O0YXfGBEGwFvLBwFvAb9eyx9wEfknlXlhdWKYlKrFG6LE/Document%20with%20spaces.tmp
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
Via: 1.1 BN1302____PAP237 (wls-colorado)

After this, I'm making another request to the URL from the Location header:

GET https://public.bl3301.livefilestore.com/y3m9aaOZd0wyyDVSJDL_DI_v0K1mU8NwXMjZ37sB6bOF7_Vq2L4AwcfuqFgO3IeJdZkp_U1q9Dq7O26O0lXJrExCDzXcv1kmex-tapAWQ8M6v298Mb7_jVzda2YBZnTTwtlCHVSY95mM6h1W7mVWaESLczOzfUTyHbP3dui4OSu8f0co8M_4TyE4w6poveTN3O0YXfGBEGwFvLBwFvAb9eyx9wEfknlXlhdWKYlKrFG6LE/Document with spaces.tmp
Authorization: bearer EwCA...ZxAQ==    (SAME AS BEFORE)
Range: bytes=0-2097152

Only this time, I'm getting a 403 Forbidden response:

StatusCode: 403
X-MSNSERVER: BL3301____PAP212
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Virus-Infected: ScanError
X-MSDAVEXT_Error: 9830409
X-QosStats: {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}
X-ThrowSite: 1e99.cd8a
X-ClientErrorCode: VirusError
X-AsmVersion: UNKNOWN; 19.22.0.0
X-MSEdge-Ref: Ref A: 33218739EA894F05BC2F8FF1CAC09ACE Ref B: 1755B82AA6FC9880B766E46DD4BB5FF8 Ref C: Wed Aug 05 06:35:30 2015 PST
Accept-Ranges: bytes
Date: Wed, 05 Aug 2015 13:35:29 GMT
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
Server: Microsoft-IIS/8.5

Trying to download the file manually from OneDrive gives me a warning message, saying "Unable to scan Document with%20spaces.tmp for viruses".

ificator commented 9 years ago

Hi @saguiitay, does this still repro for you?

saguiitay commented 9 years ago

Yes, this is still happening.

URL:

https://public.bl3301.livefilestore.com/y3mh67UfVK7UMe8I4-3li2FPGDa6dW6GRvKT8Kg8jh1MzvR1VnYf1v6HL1E1O1dCxVcVXHAgEkShMAc9ALeGJtZQAJKjWqH2osB7arapuvzJGR71GV_I9jXlPACTQNcnG8oB32bZtlAS874kQbYml4ZGP3PkMU00iILUVvQH5MNGOfJyvAwtSMml01BGaaygxlhq1AKysSFtJrwg7Wr6JL9LlTaBQiheAUUyUJGjThdo6Q/Document with spaces.tmp
ificator commented 9 years ago

Thanks for confirming - we'll need to investigate this one a little more to figure out what's going wrong. Do you see this occurring on other files as well, or is this the only one?

saguiitay commented 9 years ago

Not sure if you guys changes anything, but this seems to be working fine now...

ificator commented 9 years ago

It looks transient - I think it works for a while until some event occurs that makes it fail consistent for a period of time. Definitely a strange issue... I'll make sure it gets redirected to the right people.

batteryshark commented 9 years ago

This is happening for me as well - I think it has to do with the first file in chunks looking like a valid archive that OneDrive couldn't scan and thus thinks is malware. Is there some kind of override param like with Google Drive (acknowledgeAbuse) that allows you to download it anyway?

saguiitay commented 8 years ago

I'm afraid I'm getting this issue again. Same file:

GET https://public.by3302.livefilestore.com/y3m1AflA9ZVa7XqoHrxITSRrpDvHpz-Rj_PhefGL4st4E8BZC9p4wHZZt6jHCCV1b-40AAf4e2oZetgeMeHsXY2sVM5X6M3hWoBLET6sj6pxhQry1Mwxc8ckHxrhoAZIhtQyC7RzWTy517jjDqso5e7VDHbpyKtuLsukx5_XtRd_HEO20Ku0sb8OjwlseYm2tRGXHuxnw2aqOQpofq4UXdBk9chvyXBbVr1Avqnrpk2LiI/Document%20with%20spaces.tmp HTTP/1.1
Range: bytes=0-42649
Host: public.by3302.livefilestore.com
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

And the response I'm getting is:

HTTP/1.1 403 Forbidden
Content-Length: 0
Accept-Ranges: bytes
Server: Microsoft-IIS/8.5
P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo"
X-MSNSERVER: BY3302____PAP016
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Virus-Infected: ScanError
X-MSDAVEXT_Error: 9830409
X-QosStats: {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}
X-ThrowSite: 1e99.cd8a
X-ClientErrorCode: VirusError
X-AsmVersion: UNKNOWN; 19.34.0.0
X-MSEdge-Ref: Ref A: 3D629D7DE4A545A39F6EDDEE867A4FEF Ref B: 8D134593FB2216F1DA8144C3A5890DAD Ref C: Tue Feb 02 01:04:19 2016 PST
Date: Tue, 02 Feb 2016 09:04:19 GMT

Any chance this is related to X-ClientErrorCode: VirusError?

saguiitay commented 8 years ago

It appears that the API cannot download files that are not scanned:

image

Is there any flag that I can set in the request to automatically accept the above warning?

saguiitay commented 8 years ago

Apparently, adding AVOverride=1 to the requested URL works.

I had to look at the requests performed by the browser, since this is not documented anywhere.

ificator commented 8 years ago

Just to update this, we have tracked down a class of issues relating to an issue with the virus scanner and we're working on change that should negate the need to use AVOverride=1.

ificator commented 8 years ago

There have been a few AV related fixes that have shipped recently, given that has anyone had this issue occur in the past couple of months?

StraightfaceStudios commented 8 years ago

We are encountering this with the Cloudberry Backup application. 

2016-07-25 19:00:31,120 [Base] [19] ERROR - 
CloudBerryLab.Base.HttpUtil.Light.LightWebException
The remote server returned an error: (403) Forbidden.
   at po.A(pj )
   at po.B(pj )
System.Net.WebException
The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.GetResponse()
   at po.A(pj )

2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-MSNSERVER: BN1304____PAP175
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header Strict-Transport-Security: max-age=31536000; includeSubDomains
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-Virus-Infected: ScanError
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-MSDAVEXT_Error: 9830409
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-QosStats: {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42}
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-ThrowSite: 1e99.cd8a
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-ClientErrorCode: VirusError
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-AsmVersion: UNKNOWN; 19.46.0.0
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header X-MSEdge-Ref: Ref A: 5F6B4B4BE17D4409AECB633D4FA3697D Ref B: C945FD480CCA130AE398A0448A5F43B0 Ref C: Mon Jul 25 12:00:38 2016 PST
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header Accept-Ranges: bytes
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header Content-Length: 0
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header Date: Mon, 25 Jul 2016 19:00:38 GMT
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header P3P: CP="####"
2016-07-25 19:00:31,120 [Base] [19] ERROR - Header Server: Microsoft-IIS/8.5
2016-07-25 19:00:31,135 [CL] [19] ERROR - Command::Run failed:
Copy; Source:/####01/IMG_1742.JPG; Destination:####/IMG_1742.JPG:/20151217002833/
CloudBerryLab.Base.Exceptions.Status403Exception: The remote server returned an error: (403) Forbidden. ---> CloudBerryLab.Base.HttpUtil.Light.LightWebException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.GetResponse()
   at po.A(pj )
   --- End of inner exception stack trace ---
   at po.A(pj )
   at po.B(pj )
   --- End of inner exception stack trace ---
   at po.B(pj )
   at po.b(pj )
   at po.C(pj )
   at pp.HL(pj )
   at oI.A(op )
   at oI.A(String , QA , ICancelable )
   at Se.hA(Int64 , Int64 )
   at Se.a()
   at oI.a(String , QA , ICancelable )
   at ok.A(QA , rZ )
   at th.Gv(tA , String , TT )
   at TV.DQ()
   at TU.hV()
2016-07-25 19:00:31,135 [PL] [19] INFO  - Search thread continued
2016-07-25 19:00:31,135 [PL] [19] ERROR - Copy creator interrupted
2016-07-25 19:00:31,151 [PL] [19] INFO  - Cloud operations has been canceled
2016-07-25 19:00:31,213 [Base] [6] INFO  - The request was canceled
2016-07-25 19:00:31,213 [PL] [19] FATAL - Fatal error occurred during Upload operation. Cloud path: ###/IMG_1742.JPG:/20151217002833/IMG_1742.JPG. IsSimple: False. Modified date: 12/17/2015 12:28:33 AM. Size: 4.2 MB (4374049)
CloudBerryLab.Base.Exceptions.Status403Exception
The remote server returned an error: (403) Forbidden.
   at po.B(pj )
   at po.b(pj )
   at po.C(pj )
   at pp.HL(pj )
   at oI.A(op )
   at oI.A(String , QA , ICancelable )
   at Se.hA(Int64 , Int64 )
   at Se.a()
   at oI.a(String , QA , ICancelable )
   at ok.A(QA , rZ )
   at th.Gv(tA , String , TT )
   at TV.DQ()
   at TU.hV()
   at uL.D()
CloudBerryLab.Base.HttpUtil.Light.LightWebException
The remote server returned an error: (403) Forbidden.
   at po.A(pj )
   at po.B(pj )
System.Net.WebException
The remote server returned an error: (403) Forbidden.
   at System.Net.HttpWebRequest.GetResponse()
   at po.A(pj )

2016-07-25 19:00:31,213 [CL] [19] INFO  - The queue worker thread 19 finished.
2016-07-25 19:00:31,213 [CL] [6] ERROR - Command::Run failed:
Copy; Source:/####/IMG_1746.JPG:/20151217002748/
The operation was canceled.
ificator commented 8 years ago

Is anyone still seeing issues like this? There have been a few fixes in relation to virus scanning that hopefully resolved such repros.

StraightfaceStudios commented 7 years ago

Still getting this error.

2017-02-22 12:15:05,764 [Base] [4] ERROR - Header X-MSNSERVER: BN2BAP25420FC5B 2017-02-22 12:15:05,771 [Base] [4] ERROR - Header Strict-Transport-Security: max-age=31536000; includeSubDomains 2017-02-22 12:15:05,778 [Base] [4] ERROR - Header X-MSDAVEXT_Error: 9830409 2017-02-22 12:15:05,785 [Base] [4] ERROR - Header X-Virus-Infected: ScanError 2017-02-22 12:15:05,791 [Base] [4] ERROR - Header X-QosStats: {"ApiId":0,"ResultType":2,"SourcePropertyId":0,"TargetPropertyId":42} 2017-02-22 12:15:05,798 [Base] [4] ERROR - Header X-ThrowSite: 1e99.cd8a 2017-02-22 12:15:05,810 [Base] [4] ERROR - Header X-ClientErrorCode: VirusError 2017-02-22 12:15:05,817 [Base] [4] ERROR - Header X-AsmVersion: UNKNOWN; 18.1.0.0 2017-02-22 12:15:05,823 [Base] [4] ERROR - Header X-MSEdge-Ref: Ref A: DE3D1241C7C54CF29A9FA5025F9F37C9 Ref B: WSTEDGE0409 Ref C: Wed Feb 22 12:15:05 2017 PST 2017-02-22 12:15:05,829 [Base] [4] ERROR - Header Accept-Ranges: bytes 2017-02-22 12:15:05,835 [Base] [4] ERROR - Header Content-Length: 0 2017-02-22 12:15:05,842 [Base] [4] ERROR - Header Date: Wed, 22 Feb 2017 20:15:05 GMT 2017-02-22 12:15:05,850 [Base] [4] ERROR - Header P3P: CP="BUS CUR CONo FIN IVDo ONL OUR PHY SAMo TELo" 2017-02-22 12:15:05,857 [Base] [4] ERROR - Header Server: Microsoft-IIS/8.5 2017-02-22 12:15:05,881 [CL] [4] ERROR - Command::Run failed:

File: /IPKs/ventana/kernel-dev_3.10.17-12_ventana.ipk; Destination:ventana (Embedded system image)

StraightfaceStudios commented 7 years ago

6 months later... still getting that error on a load of files.

ificator commented 7 years ago

@StraightfaceStudios It shouldn't be possible for that error to be returned any more. If you're still seeing it could you provide a new set of response data similar to what you provided above?

StraightfaceStudios commented 7 years ago

Unfortunately I had just deleted the job to "start from scratch" and of course now everything is working perfectly without any errors and of course it deleted the logs for the previous job when I deleted the job. #EveryTime.

If you say the error is dead I'll believe you. Maybe I accidentally missed the end of the previous logs from months ago.

Thanks though for the follow up! I'll post specific logs if through some horrible miracle it starts breaking again.

ificator commented 7 years ago

Thanks @StraightfaceStudios, definitely let us know if you see something like this again. In the past a ScanError would fail the request, but we made a change to make sure that doesn't happen so hopefully it's resolved :).