search

WayneJz / teslamate-addr-fix

To fix teslamate broken addresses caused by openstreetmap unavailability
MIT License
68 stars 7 forks source link

err=osm returns error status, status=403 #19

Closed Becase closed 3 months ago

Becase commented 4 months ago

作者你好，感谢你提供的修复工具。

我的代理地址：http://192.168.1.88:18888

不添加代理curl

执行curl -v "https://nominatim.openstreetmap.org/reverse?lat=19.913121&lon=110.316397&format=json" 输出如下： `

添加代理curl

部署环境用代理可以curl到地址，如下： curl -v "https://nominatim.openstreetmap.org/reverse?lat=19.913121&lon=110.316397&format=json" -x http://192.168.1.88:18888 输出日志如下： `

CONNECT nominatim.openstreetmap.org:443 HTTP/1.1 Host: nominatim.openstreetmap.org:443 User-Agent: curl/7.86.0 Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established <

  • TLSv1.0 (OUT), TLS header, Certificate Status (22):

  • TLSv1.3 (OUT), TLS handshake, Client hello (1):

  • TLSv1.2 (IN), TLS header, Certificate Status (22):

  • TLSv1.3 (IN), TLS handshake, Server hello (2):

  • TLSv1.2 (IN), TLS header, Finished (20):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, Certificate (11):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, CERT verify (15):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, Finished (20):

  • TLSv1.2 (OUT), TLS header, Finished (20):

  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

  • TLSv1.3 (OUT), TLS handshake, Finished (20):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23): GET /reverse?lat=19.913121&lon=110.316397&format=json HTTP/2 Host: nominatim.openstreetmap.org user-agent: curl/7.86.0 accept: /

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • TLSv1.2 (IN), TLS header, Supplemental data (23):

  • TLSv1.2 (OUT), TLS header, Supplemental data (23):

  • TLSv1.2 (IN), TLS header, Supplemental data (23): < HTTP/2 200 < server: nginx < date: Wed, 05 Jun 2024 16:40:16 GMT < content-type: application/json; charset=utf-8 < content-length: 613 < {"place_id":237646966,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright","osm_type":"way","osm_id":220068262,"lat":"19.91281585","lon":"110.31585813238507","class":"amenity","type":"parking","place_rank":30,"importance":9.99999999995449e-06,"addresstype":"amenity","name":"","display_name":"羊山大道, 龙华区, 海口市, 海南省, 571155, 中国","address":{"road":"羊山大道","city":"龙华区","state":"海南省","ISO3166-2-lvl4":"CN-HI","postcode":"571155","country":"中国","country_code":"cn"},"boundingbox":["19.9119490","19.9136180","110.3150186","110.3166638"]}`

teslamate-addr-fix加代理执行提示status=403

./teslamate-addr-fix -db myteslamate -host 172.17.0.1 -password password -user teslamate -port 5432 -proxy http://192.168.1.88:18888 -timeout 15 执行后提示： psql_client.go:233: get address from osm failed, lat=xx, lon=xx, err=osm returns error status, status=403, url=https://nominatim.openstreetmap.org/reverse?lat=xx&lon=xx&format=json

请问这是啥原因呢？

WayneJz commented 4 months ago

你好，试试把命令行参数quote起来再运行一下？ lat=xx, lon=xx应该是你隐去了吧，可以再看看报错这行，实际的经纬度格式是否正确？

Becase commented 4 months ago

你好，试试把命令行参数quote起来再运行一下？ lat=xx, lon=xx应该是你隐去了吧，可以再看看报错这行，实际的经纬度格式是否正确？

请问怎么quote，我将代理地址增加引号后尝试执行： ./teslamate-addr-fix -db myteslamate -host 172.17.0.1 -password password -user teslamate -port 5432 -proxy “http://192.168.1.88:18888” -timeout 15

输出：psql_client.go:193: get address from osm failed, lat=xx.xxxx65, lon=xx.xxxx84, err=osm returns error status, status=403, url=https://nominatim.openstreetmap.org/reverse?lat=xx.xxxx65&lon=xx.xxxx84&format=json

WayneJz commented 4 months ago 
./teslamate-addr-fix -db myteslamate -host '172.17.0.1' -password password  -user teslamate -port 5432 -proxy 'http://192.168.1.88:18888' -timeout 15

试试这样

Becase commented 4 months ago

应该是go http.Client默认的user-agent被openstreetmap阻止访问了

使用curl默认的user-agent访问

curl "https://nominatim.openstreetmap.org/reverse?lat=19.913121&lon=110.316397&format=json" -x http://192.168.1.88:18888 --user-agent "curl/[7.86.0]"

{"place_id":237646966,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright","osm_type":"way","osm_id":220068262,"lat":"19.91281585","lon":"110.31585813238507","class":"amenity","type":"parking","place_rank":30,"importance":9.99999999995449e-06,"addresstype":"amenity","name":"","display_name":"羊山大道, 龙华区, 海口市, 海南省, 571155, 中国","address":{"road":"羊山大道","city":"龙华区","state":"海南省","ISO3166-2-lvl4":"CN-HI","postcode":"571155","country":"中国","country_code":"cn"},"boundingbox":["19.9119490","19.9136180","110.3150186","110.3166638"]}

使用go默认的user-agent curl

curl "https://nominatim.openstreetmap.org/reverse?lat=19.913121&lon=110.316397&format=json" -x http://192.168.1.88:18888 --user-agent "Go-http-client/1.1"

<html>
<head>
<title>Access blocked</title>
</head>
<body>
<h1>Access blocked</h1>

<p>You have been blocked because you have violated the
<a href="https://operations.osmfoundation.org/policies/nominatim/">usage policy</a>
of OSM's Nominatim geocoding service. Please be aware that OSM's resources are
limited and shared between many users. The usage policy is there to ensure that
the service remains usable for everybody.</p>

<p>Please review the terms and make sure that your
software adheres to the terms. You should in particular verify that you have set a
<b>custom HTTP referrer or HTTP user agent</b> that identifies your application, and
that you are not overusing the service with massive bulk requests.</p>

<p>If you feel that this block is unjustified or remains after you have adopted
your usage, you may contact the Nominatim system administrator at
nominatim@openstreetmap.org to have this block lifted.</p>
</body>
</head>

使用chrome user-agent curl,结果正确

curl "https://nominatim.openstreetmap.org/reverse?lat=19.913121&lon=110.316397&format=json" -x http://192.168.1.88:18888 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"

{"place_id":237646966,"licence":"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright","osm_type":"way","osm_id":220068262,"lat":"19.91281585","lon":"110.31585813238507","class":"amenity","type":"parking","place_rank":30,"importance":9.99999999995449e-06,"addresstype":"amenity","name":"","display_name":"羊山大道, 龙华区, 海口市, 海南省, 571155, 中国","address":{"road":"羊山大道","city":"龙华区","state":"海南省","ISO3166-2-lvl4":"CN-HI","postcode":"571155","country":"中国","country_code":"cn"},"boundingbox":["19.9119490","19.9136180","110.3150186","110.3166638"]}

这也就是为什么我的代理可以curl到，但是teslamate-addr-fix无法获取地址的原因吧。

Becase commented 3 months ago 
const (
    osmReverseURL   = "https://nominatim.openstreetmap.org/reverse?lat=%.6f&lon=%.6f&format=json"
    chromeUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
)
func getAddressByProxy(latitude, longitude float64) (*OsmRevAddress, error) {
    if latitude == 0 && longitude == 0 {
        return nil, fmt.Errorf("arg error, latitude=0, longitude=0")
    }
    /*
        reqURL := fmt.Sprintf(osmReverseURL, latitude, longitude)

        fmt.Println("START--------------------")

        fmt.Println(reqURL)

        fmt.Println("cli is null:", cli == nil)

        rsp, err := cli.Get(reqURL)

        fmt.Println("rsp:", rsp)
        fmt.Println("err:", err)
        if err != nil {
            return nil, err
        }
    */

    reqURL := fmt.Sprintf(osmReverseURL, latitude, longitude)

    // 创建一个新的 HTTP 请求
    req, err := http.NewRequest("GET", reqURL, nil)

    if err != nil {
        fmt.Println("Error creating request:", err)
        return nil, err
    }

    // 设置 User-Agent 头部
    req.Header.Set("User-Agent", chromeUserAgent)
    // req.Header.Set("User-Agent", curlUserAgent)
    // req.Header.Set("User-Agent", golangUserAgent)

    rsp, err := cli.Do(req)

    if err != nil {
        return nil, err
    }

……