Handling "Host: localhost"

[fabled]

I am currently trying to get HP Color LaserJet FlowMFP M578 to work. And seems on USB-IPP's part things are mostly good now!

However, the AVAHI URLs reported are:

=     lo IPv4 HP Color LaserJet FlowMFP M578 [7839F2] (USB) _ipp._tcp            local
   hostname = [vostro.local]
   address = [127.0.0.1]
   port = [60001]
   txt = ["air=none" "mopria-certified=2.0" "rp=ipp/print" "priority=50" "kind=document,envelope,photo" "PaperMax=legal-A4" "URF=V1.4,CP99,RS300-600-1200,MT1-2-3-5-12,W8,PQ4,SRGB24,IS20-21,IFU21,DM1,OB10" "UUID=<removed>" "Color=T" "Duplex=T" "note=" "qtotal=1" "usb_MDL=HP Color LaserJet FlowMFP M578" "usb_MFG=Hewlett-Packard" "usb_CMD=PCL,PCLXL,POSTSCRIPT,PDF,URF,PWGRaster,TIFF" "ty=HP Color LaserJet FlowMFP M578" "product=(HP Color LaserJet FlowMFP M578)" "pdl=text/plain,application/octet-stream,image/jpeg,application/vnd.hp-PCL,application/vnd.hp-PCLXL,application/postscript,application/pdf,image/urf,image/pwg-raster,image/tiff,application/PCLm" "txtvers=1" "adminurl=http://localhost:60001/airprint_mob_status.htm" "Fax=F" "Scan=T"]
=     lo IPv4 HP Color LaserJet FlowMFP M578 [7839F2] (USB) _uscan._tcp          local
   hostname = [vostro.local]
   address = [127.0.0.1]
   port = [60001]
   txt = ["duplex=T" "is=platen,adf" "cs=binary,color,grayscale" "UUID=<removed>" "adminurl=http://localhost:60001/" "representation=/hp/device/printer128.png" "pdl=application/pdf,image/jpeg,image/tiff" "ty=HP Color LaserJet FlowMFP M578" "rs=eSCL" "vers=2.8" "txtvers=1"]

This creates the following problems:

• CUPS does not use AVAHI to resolve the host name. The hostname does is not resolvable via normal resolved. Thus it cannot connect to the printer.
• SANE uses the IP-address (which likely is a problem). But the actual HP printer requires to use "localhost" as the hostname in HTTP requests or it will give Redirect header to use it.

For the above reasons, I'm wondering if it would be possible to advertise localhost on the loopback interface as the hostname?

[alexpevzner]

ipp-usb actually advertises localhost, but Avahi replaces it with whatever you see.

This is Avahi "feature". It is a long story that lasts for at least several years.

RFC that defines DNS-SD explicitly doesn't allow to use localhost as a host name, though it is essential for correct work of IPP over USB (DNS-SD and IPP-over-USB specs are controverting at this aspect) . Many people tried to convince Avahi author to ignore this part of specification, but nobody succeed so far. So we can only speak about workarounds.

Redirect is inserted by ipp-usb, not by printer, because some printers really require Host: localhost to be set. sane-escl should follow these redirects.

I'd like also ask you to test an alternative eSCL/WSD implementation, the sane-airscan:

• https://github.com/alexpevzner/sane-airscan - project page
• https://download.opensuse.org/repositories/home:/pzz/ - binary packages for many distros

[fabled]

I tried sane-airscan, but it did not work either. Likely because this printer request different looking scan request xml. See https://gitlab.com/sane-project/backends/-/issues/424.

Is it possible to disable the auto redirect from ipp-usb?

[fabled]

Ah, found also avahi patch to allow using localhost. I'll probably apply that.

[alexpevzner]

I would appreciate if you could gather protocol trace logs with sane-airscan (I want to see the entire conversation, not only failed request).

To do so, please uncomment the following lines in the /etc/sane.d/airscan.conf file:

[debug]
trace = ~/airscan/trace
enable = true

Logs will be in the ~/airscan/trace directory. .tar files are not needed for now, only .log files.

[fabled]

scanimage-HP-Color-LaserJet-FlowMFP-M578-[7839F2]-(USB).log

As you see, the Scanner is failing in the ScanJobs request with 409. It seems to require it in some unusual format, as reported in the sane ticket.

[alexpevzner]

Yes, I see. I also wanted to look how this device sends its capabilities, status etc.

I will try to support this format of requests and will come to you back later.

[fabled]

Figured it out. Seems the HTTP Host header needs to not contain port number on the ScanJobs request, it needs to match exactly Host: localhost (for usb-ipp to pass it through, and no port for the device to accept it).

[alexpevzner]

You want to say, with Host: localhost this device scans even with the same request format, as currently present in sane-airscan?

[fabled]

I think so. At least the curl requests I made, started to work with the sane escl backend request after that. So I think it's picky about it.

But when trying to use Host: localhost in all requests, usb-ipp will give redirects to the port numbered one. Is the redirect logic for GET requests only, and bypassed for POST? I'm really hoping to turn off the redirects now from usb-ipp.

[fabled]

Ok. Found how to remove the redirect stuff... And yes, the standard sane escl works now as long "Host: localhost" is without port number. Did not test airscan yet, but I suspect it works too. It's not perfect yet as ADF scan seems to abort early, but at least great progress.

[fabled]

How about doing:

diff --git a/http.go b/http.go
index 3b1ff8f..e2760d3 100644
--- a/http.go
+++ b/http.go
@@ -148,8 +148,9 @@ func (proxy *HTTPProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        // This redirection fixes compatibility with these printers for
        // clients that follow redirects (i.e., web browser and sane-airscan;
        // CUPS unfortunately doesn't follow redirects)
-       if localAddr.IP.IsLoopback() &&
-               !strings.HasPrefix(strings.ToLower(r.Host), "localhost:") &&
+       lowercaseHost := strings.ToLower(r.Host)
+       if localAddr.IP.IsLoopback() && lowercaseHost != "localhost" &&
+               !strings.HasPrefix(lowercaseHost, "localhost:") &&
                r.Method == "GET" || r.Method == "HEAD" {

                url := *r.URL

This would fix the code to comply with the comment above. Currently it does not match "Host: localhost".

[alexpevzner]

I've slightly rewritten this code to avoid unneeded string allocation in a case we unlikely to redirect, but otherwise agreed with this patch; please retest

[alexpevzner]

And regarding the Host: localhost setting in outgoing ScanJons request, this is the same problem, as with the HP LaserJet MFP M630, and the cure is the same.

Please, rebuild and retest sane-airscan, I hope this problem will go away.

[fabled]

Ah. Brilliant. Thanks. Will test on Monday but it'll probably fix it. Since this seems generic problem on HP, how about detecting making it based on the http reply Server header, or something similarly generic?

That is, IP is loopback and Server: HP_Compact_Server

[alexpevzner]

When this problem was initially found, I've noticed, that VueScan always sets Host: localhost, and in assumption that they did much more compatibility testing, that I can, I did the same.

Later I've discovered that this hack conflicts with some other devices (see https://github.com/alexpevzner/sane-airscan/issues/92, https://github.com/alexpevzner/sane-airscan/issues/98). So I've decided to be more conservative, and enable this hack only when it is needed.

Note, even most of HP printers doesn't require this hack. So far, both problematic printers were of the enterprise class.

[alexpevzner]

I also very interested to know, is ADF problem, that you've discovered with sane-escl, exists in the sane-airscan too.

[fabled]

Yes, I understand it's an ugly hack and needs to be done selectively.

My suggestions is: For the initial GET ScannerCapabilities response, parse the HTTP headers the server responds with. The Server header is the server codebase name. The affected HP devices seem to have HP_Compact_Server, so that might be good enough check to figure out if it's the broken HP code base. And hopefully it would cover potential other current/future devices with the same bug instead of needing to quirk list them all. I could draft a patch for this if acceptable.

I will get back to the device on Monday, and retest with the updated sane-airscan.

[alexpevzner]

Honestly, as my name is not "Microsoft", so I can't perform a wide testing after each change, I prefer to be very conservative when making changes that may affect many already working devices.

Regarding this particular idea, I will keep it in mind. For now I will not make such a generic change in behaviour, but I will reconsider it if and when more devices with similar problem will be reported.

BTW, I don't think it is "bug". Somebody on HP has written this code, that checks Host: value against localhost (even in network mode, at least for HP LaserJet MFP M630) and it was done with some purpose. Unfortunately, currently I can't guess this purpose. But it effectively prevents some 3rd party software, like Mopria for Android, from access to the device.

[fabled]

I think commit 5f743231ae0e09807d15f68a92a174bda41f29aa fixed the usb-ipp issue on this.

And I created https://github.com/alexpevzner/sane-airscan/issues/116 to track the quirk handling. I understand we want to be conservative on this. So far the Service: line seems to be reliable. But understandably more is needed. Some additional ideas listed on this ticket.

I think the problem is solved on usb-ipp part now.

[fabled]

The AVAHI issue not allowing localhost is really cumbersome. When resolving the hostname.local entry, it defaults to something else than loopback IP-address. Causing all kind of other funkyness. I suppose it's simplest to patch AVAHI to allow localhost for loopback interface.

[alexpevzner]

ipp-usb part of this work released as 0.9.17