gateway subdomains: *.localhost not found

bmwiedemann commented 4 years ago

Version information:

go-ipfs version: 0.5.0 Repo version: 9 System version: amd64/linux Golang version: go1.14

openSUSE Leap 15.1

Description:

The web gateway from PR #6096 redirects to non-resolvable subdomains of localhost and I can find no documentation on how to make it resolve.

$ curl -L -v localhost:8080/ipns/opensuse.zq1.de/
[...]
< Location: http://opensuse.zq1.de.ipns.localhost:8080/
< X-Ipfs-Path: /ipns/opensuse.zq1.de/
< Date: Fri, 08 May 2020 07:44:31 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://opensuse.zq1.de.ipns.localhost:8080/'
* Could not resolve host: opensuse.zq1.de.ipns.localhost

$ host ipfs.io.ipns.localhost
Host ipfs.io.ipns.localhost not found: 3(NXDOMAIN)

$ ping ipfs.io.ipns.localhost
ping: ipfs.io.ipns.localhost: Name or service not known

RubenKelevra commented 4 years ago

Golang version: go1.14

Sidenote: golang 1.14 up to 1.14.1 had some bugs IPFS is going to trigger. Either update to golang >= 1.14.2 or downgrade to 1.13.8.

RubenKelevra commented 4 years ago

I can't confirm this bug under ArchLinux - IPFS running on port 80 here (which shouldn't matter much).

curl

[ruben@i3 ~]$  curl -L -v http://localhost/ipns/ipfs.io/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:80...
* Connected to localhost (::1) port 80 (#0)
> GET /ipns/ipfs.io/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Accept-Ranges: bytes
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Clear-Site-Data: "cookies", "storage"
< Content-Length: 31600
< Content-Type: text/html
< Etag: "bafybeidhycjoffkqxrzx3gutcf5quq3tczkikxfsocdayt6o3lzqiqqj4q"
< Last-Modified: Fri, 08 May 2020 11:58:31 GMT
< Location: http://ipfs.io.ipns.localhost/
< X-Ipfs-Path: /ipns/ipfs.io/
< Date: Fri, 08 May 2020 11:58:31 GMT
< 
* Ignoring the response-body
{ [3349 bytes data]
100 31600  100 31600    0     0   482k      0 --:--:-- --:--:-- --:--:--  482k
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://ipfs.io.ipns.localhost/'
*   Trying ::1:80...
* Connected to ipfs.io.ipns.localhost (::1) port 80 (#1)
> GET / HTTP/1.1
> Host: ipfs.io.ipns.localhost
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Content-Length: 31600
< Content-Type: text/html
< Etag: "bafybeidhycjoffkqxrzx3gutcf5quq3tczkikxfsocdayt6o3lzqiqqj4q"
< Last-Modified: Fri, 08 May 2020 11:58:31 GMT
< X-Ipfs-Path: /ipns/ipfs.io/
< Date: Fri, 08 May 2020 11:58:31 GMT
< 
{ [3445 bytes data]
100 31600  100 31600    0     0   203k      0 --:--:-- --:--:-- --:--:-- 26.8M
* Connection #1 to host ipfs.io.ipns.localhost left intact

ping

[ruben@i3 ~]$ ping ipfs.io.ipns.localhost -c 4
PING ipfs.io.ipns.localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.052 ms
64 bytes from localhost (::1): icmp_seq=3 ttl=64 time=0.033 ms
64 bytes from localhost (::1): icmp_seq=4 ttl=64 time=0.067 ms

--- ipfs.io.ipns.localhost ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3077ms
rtt min/avg/max/mdev = 0.028/0.045/0.067/0.015 ms

RubenKelevra commented 4 years ago

Same result with a new initialized daemon, I've just added ::1 as additionally listening address for the web-gateway:

[ruben@i3 ~]$ curl -L -v http://localhost:8080/ipns/ipfs.io/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
> GET /ipns/ipfs.io/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Accept-Ranges: bytes
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Clear-Site-Data: "cookies", "storage"
< Content-Length: 31600
< Content-Type: text/html
< Etag: "bafybeidhycjoffkqxrzx3gutcf5quq3tczkikxfsocdayt6o3lzqiqqj4q"
< Last-Modified: Fri, 08 May 2020 12:09:15 GMT
< Location: http://ipfs.io.ipns.localhost:8080/
< X-Ipfs-Path: /ipns/ipfs.io/
< Date: Fri, 08 May 2020 12:09:16 GMT
< 
* Ignoring the response-body
{ [31600 bytes data]
100 31600  100 31600    0     0  42702      0 --:--:-- --:--:-- --:--:-- 42645
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://ipfs.io.ipns.localhost:8080/'
*   Trying ::1:8080...
* Connected to ipfs.io.ipns.localhost (::1) port 8080 (#1)
> GET / HTTP/1.1
> Host: ipfs.io.ipns.localhost:8080
> User-Agent: curl/7.70.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Access-Control-Allow-Headers: Content-Type
< Access-Control-Allow-Headers: Range
< Access-Control-Allow-Headers: User-Agent
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: GET
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Range
< Access-Control-Expose-Headers: X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output
< Content-Length: 31600
< Content-Type: text/html
< Etag: "bafybeidhycjoffkqxrzx3gutcf5quq3tczkikxfsocdayt6o3lzqiqqj4q"
< Last-Modified: Fri, 08 May 2020 12:09:16 GMT
< X-Ipfs-Path: /ipns/ipfs.io/
< Date: Fri, 08 May 2020 12:09:16 GMT
< 
{ [3445 bytes data]
100 31600  100 31600    0     0  37352      0 --:--:-- --:--:-- --:--:-- 37352
* Connection #1 to host ipfs.io.ipns.localhost left intact

lidel commented 4 years ago

@RubenKelevra lack of support for *.localhost subdomains is a known issue on some linux distributions, unfortunately :(

@bmwiedemann please fill an upstream issue in openSUSE

Until its fixed there, here are some workarounds:

use dnsmasq as your DNS server with custom config
use Gateway port as HTTP Proxy in your HTTP client
- (curl specific) on top of proxy you may need to manually provide DNS mapping curl -L -x http://127.0.0.1:8080 --resolve localhost:8080:127.0.0.1 http://localhost:8080/ipns/ipfs.io/
(in non-browser contexts, where you don't need Origin isolation) it is possible to use http://127.0.0.1:8080 as old-school path gateway

@Stebalien what would be the best place to add above troubleshooting notes?

RubenKelevra commented 4 years ago

@RubenKelevra lack of support for *.localhost subdomains is a known issue on some linux distributions, unfortunately :(

@bmwiedemann please fill an upstream issue in openSUSE

Interesting, especially since ArchLinux tries always to use the minimum amount of patching/configuration.

bmwiedemann commented 4 years ago

@bmwiedemann please fill an upstream issue in openSUSE

I am an openSUSE developer, but to get it working I still need to understand why it works for others. E.g. here I have

$ grep hosts /etc/nsswitch.conf
hosts:          files dns

$ grep localhost /etc/hosts
127.0.0.1       localhost
::1             localhost ipv6-localhost ipv6-loopback

and the normal external resolvers in /etc/resolv.conf

Stebalien commented 4 years ago

Some browsers (should include firefox and chrome?) will automatically resolve *.localhost domains internally. What browser are you using?
Are you using a local DNS resolver? Most (including systemd-resolved) name servers will automatically resolve all *.localhost domains to 127.0.0.1.
nss-myhostname and nss-resolved will also resolve *.localhost.

TL;DR: Adding myhostname to your nsswitch and/or using systemd-resolved should fix it.

bmwiedemann commented 4 years ago

That server has no browsers installed. I was using curl -L. Got it working thus:

install nss-myhostname package (will be pulled in as dependency of go-ipfs package soon)
edit /etc/nsswitch.conf to have hosts: myhostname files dns (should be documented somewhere)
service nscd restart

Stebalien commented 4 years ago

Ah, I see... we discussed special-casing curl and just not redirecting for curl. Note: curl without -L should still magically work, but your solution of nss-myhostname is definitely a better solution.

Stebalien commented 4 years ago

I've filed https://github.com/ipfs/go-ipfs/issues/7293 to summarize this issue. LGTY?

RubenKelevra commented 4 years ago

@bmwiedemann

just for the record, my machine uses:

[ruben@i3 ~]$ grep localhost /etc/hosts
127.0.0.1       localhost
::1             localhost
[ruben@i3 ~]$ grep hosts /etc/nsswitch.conf
hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns

Stebalien commented 3 years ago

@lidel has the browser story changed here?

bmwiedemann commented 3 years ago

On openSUSE, I added nss-myhostname as a runtime requirement for go-ipfs and then a sysadmin only has to add myhostname to /etc/nsswitch.conf

There will always be plenty different HTTP clients (not only browsers), so NSS is the only reliable way to solve this for all of them.

lidel commented 3 years ago

When it comes to browser vendors, our collab with Igalia contributed to fixing this in Firefox 84 (bug 1220810). This means both Firefox and Chromium skip DNS lookup and *.localhost names work as expected on every platform. Safari is tracked in ~~but 21898~~ https://bugs.webkit.org/show_bug.cgi?id=160504

Hopefully over time, the basic sanity suggested in RFC6761#section-6.3 will get implemented at the operating system level.

Until then:

each user agent needs to fix this in user land (like Firefox and Chromium did)
or user/OS needs to reconfigure their DNS resolver (setting up nss-myhostname on Linux is an example of this)
or user needs to set up IPFS gateway port as HTTP proxy (so name is passed with request, and no DNS lookup is triggered)

xrdavies commented 3 years ago

With go-ipfs version: 0.9.0 on mac, wget reports the same issue.

montanaflynn commented 3 years ago

ipfs version: Version 0.17.0 (0.17.0)

The request to:

http://localhost:8080/ipfs/QmeSjSinHpPnmXmspMjwiXyN6zS4E9zccariGR3jxcaWtq/1

Redirects to

http://bafybeihpjhkeuiq3k6nqa3fkgeigeri7iebtrsuyuey5y6vy36n345xmbi.ipfs.localhost:8080/1

I can access it in chrome, but not through wget, curl or my go programs.

Is there a way to just disable the redirect?

lidel commented 3 years ago

Is there a way to just disable the redirect?

Yes! :point_right: With tools like curl or wget use URLs with the local IP (http://127.0.0.1:8080/ipfs/..). It does not require DNS lookup, will work on macOS, and the response returned by go-ipfs won't redirect to a subdomain.

Subdomain redirect on localhost really matters only in a web browser context. It ensures websites and web apps loaded from IPFS get proper origin isolation, and that the URL root will match the content root (so static website generators "just work").

montanaflynn commented 3 years ago

Hi @lidel,

Thanks! I ended up figuring it out by changing my ipfs settings:

    "PublicGateways": {
      "localhost": {
        "Paths": ["/ipfs"],
        "UseSubdomains": false
      }
    },

I'm only using the ipfs for my programs and not accessing them in the browser so it works for me, but I'll keep using local IP in mind if that changes.

livid commented 2 years ago

@lidel

I followed the Safari bug link you mentioned, but it seems not about the issue.

After a little bit of search, I found this:

https://bugs.webkit.org/show_bug.cgi?id=160504

It seems not moving too much there :(

lidel commented 1 year ago

I am closing this as there is nothing we can do on the Kubo side, and we don't have anything better than *.localhost at the moment (see window.isSecureContext notes in https://github.com/ipfs/in-web-browsers/issues/109).

These are issues with DNS resolvers of some operating systems, actions to take are:

macOS: raise support in https://bugs.webkit.org/show_bug.cgi?id=160504
Linux distributions which ship with resolver that does not resolve *.localhost: fill issue in respective repo of your distribution.

TLDR, quick fixes

If you only need *.localhost to work in a browser, switching to Chromium or Firefox will do the trick.
If you dont care about browsers and window.isSecureContext, and only use gateway for curl, CLI etc, use http://127.0.0.1:<gwport> instead of http:///localhost:<gwport> – gateway on IP acts like a path gateway and will not redirect you to non-existing subdomain.

ipfs / kubo