[Bug] DuckDuckGo fails to download a file from one-time links

starius commented 1 week ago

Describe the bug

I have a site which hosts one-time links: https://pastacity.nl/?tab=file After a file (e.g. PDF) is uploaded, it can be downloaded only once, by design of the site. (This is used a simple solution to share with some degree of privacy.)

This approach works well in many browsers, e.g. Firefox and Chrome both on PC and on mobile, but DuckDuckGo fails to download a file from such a link. The reason is that it sends two HTTP requests, both GET. The first GET request causes the link to expire and the second request (actual file download) is not working:

GET /fortune HTTP/1.1
Host: 1.2.3.4:8042
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive
Referer: http://95.216.185.106:8042/api/create
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Mobile Safari/537.36
X-Requested-With: com.duckduckgo.mobile.android

GET /fortune HTTP/1.1
Host: 1.2.3.4:8042
Accept-Encoding: gzip
Connection: Keep-Alive
Cookie: 
User-Agent: ddg_android/5.216.2 (com.duckduckgo.mobile.android; Android API 34)

I think, DuckDuckGo should send the first request as HEAD, because it is a probing request, not expecting to download real data. Then my site (and similar sites) can distinguish the first request as a probe and to prevent data expiry so the second (real) request can successfully download it.

This issue was originally reported to the repo hosting the site: https://github.com/starius/pasta/issues/8#issuecomment-2035362389

How to Reproduce

In DuckDuckGo on Android:

Go to https://pastacity.nl/?tab=file
Select a PDF file
Click on Upload button
Click on the created link
Click on "Save to Downloads" in the popup window
File download failed.

Expected behavior

The file is downloaded back to your phone. (This actually works in other browsers.)

Environment

- DDG App Version: 5.216.2 (52162000)
- Device: Galaxy A73 5G
- OS: Android 14

github-actions[bot] commented 1 week ago

Thank you for opening an Issue in our Repository. The issue has been forwarded to the team and we'll follow up as soon as we have time to investigate. As stated in our Contribution Guidelines, requests for feedback should be addressed via the Feedback section in the Android app.

cmonfortep commented 3 days ago

I've been testing this scenario, and you are right, there are 2 /get request. However, not sure about the strategy to fix this.

First one corresponds to just loading the site as a normal website: User submits the url or clicks a link, we trigger the loading site logic. We use webview, so this is just webview.loadURL.
Second one corresponds to the actual file download: Webview has a callback to notify us that user is trying to download a file and provides us the url, mimetype, etc. so here we trigger the actual download.

My assumption is we are limited to the webview api here compared to other browsers. Will share this internally in case we can think about an approach.

starius commented 2 days ago

@cmonfortep Thanks for the response!

I understand now, why sending a HEAD request is not a good idea: in case of HTLM or image content it would cause an additional GET request to load the content.

What would you recommend to implement on the server side to make the site working in DuckDuckGO? Would the following approach work?

distinguish the first request (made by WebView) and return zeros or random data of the same length. Don't expire the URL.
on the second request (made by ddg host app) return real data and expire the URL

This seems as ad-hoc solution and it will fail in the future, if DDG starts downloading the data from the first request. However I don't see a better approach. What do you think?

What is a good way to distinguish the first request? Maybe header X-Requested-With containing "duckduckgo" substring?

duckduckgo / Android