search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123k stars 17.53k forks source link

affected/package: encoding/json can not encoding Http Response entity #50307

Closed ghost closed 2 years ago

ghost commented 2 years ago

What version of Go are you using (go version)?

$ go version
1.17.5

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/nomore/Library/Caches/go-build"
GOENV="/Users/nomore/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/nomore/go/pkg/mod"
GONOPROXY=""
GONOSUMDB="*"
GOOS="darwin"
GOPATH="/Users/nomore/go"
GOPRIVATE=""
GOPROXY="https://goproxy.cn/"
GOROOT="/Users/nomore/.asdf/installs/golang/1.17.5/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/nomore/.asdf/installs/golang/1.17.5/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s8/w2xf8qxj6v712bq3_w_p7clw0000gp/T/go-build3423845732=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

     // Click here and start typing.
package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    hc, err := http.NewRequest("GET", "https://gitlab.com/api/v4/templates/gitignores", bytes.NewBuffer([]byte{}))
    if err != nil {
        panic(err)
    }

    clt := http.Client{}
    response, err := clt.Do(hc)
    jsonStr, err := json.Marshal(response)
    fmt.Printf("Response: '%+v' \n", string(jsonStr))

    fmt.Printf("Response raw: '%+v'\n", response)

    bodyAll, err := ioutil.ReadAll(response.Body)

    fmt.Println("response status: " + response.Status)
    fmt.Println("body " + string(bodyAll))
}

What did you expect to see?

fmt.Printf("Response: '%+v' \n", string(jsonStr)) return the formatted JSON string.

What did you see instead?

fmt.Printf("Response: '%+v' \n", string(jsonStr))

Empty string

For compare, With Java , we can easy to dump all response object into a JSON String

ghost commented 2 years ago

for my local test

 $ go run main.go
Response: ''
Response raw: '&{Status:200 OK StatusCode:200 Proto:HTTP/2.0 ProtoMajor:2 ProtoMinor:0 Header:map[Cache-Control:[max-age=0, private, must-revalidate] Cf-Cache-Status:[DYNAMIC] Cf-Ray:[6c1773661c097d0a-LAX] Content-Type:[application/json] Date:[Wed, 22 Dec 2021 06:56:31 GMT] Etag:[W/"0d495ac44aeb780f688f3eb8674fdbfc"] Expect-Ct:[max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"] Gitlab-Lb:[fe-17-lb-gprd] Gitlab-Sv:[localhost] Link:[<https://gitlab.com/api/v4/templates/gitignores?page=2&per_page=20>; rel="next", <https://gitlab.com/api/v4/templates/gitignores?page=1&per_page=20>; rel="first", <https://gitlab.com/api/v4/templates/gitignores?page=10&per_page=20>; rel="last"] Nel:[{"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}] Ratelimit-Limit:[2000] Ratelimit-Observed:[1] Ratelimit-Remaining:[1999] Ratelimit-Reset:[1640156251] Ratelimit-Resettime:[Wed, 22 Dec 2021 06:57:31 GMT] Referrer-Policy:[strict-origin-when-cross-origin] Report-To:[{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2B%2Fsodq%2FpvR8ELbT%2FxjoW0XuETKakT2TUIYH2gQzVRH6VeKyE%2B%2FCOjN2N0K9SHqaL9Kt3dpB0djL%2FYaiDP7V8kDFol%2BTDSqjBuLkZf%2FPgeN417zgcMEtjVg0pe2Q%3D"}],"group":"cf-nel","max_age":604800}] Server:[cloudflare] Strict-Transport-Security:[max-age=31536000] Vary:[Origin] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Next-Page:[2] X-Page:[1] X-Per-Page:[20] X-Prev-Page:[] X-Request-Id:[01FQGETQ3KPJ9KE1TC0M5KX4J8] X-Runtime:[0.020044] X-Total:[191] X-Total-Pages:[10]] Body:0xc0002d29c0 ContentLength:-1 TransferEncoding:[] Close:false Uncompressed:true Trailer:map[] Request:0xc0000f4000 TLS:0xc00031e0b0}'
response status: 200 OK
body [{"key":"Actionscript","name":"Actionscript"},{"key":"Ada","name":"Ada"},{"key":"Agda","name":"Agda"},{"key":"Android","name":"Android"},{"key":"AppEngine","name":"AppEngine"},{"key":"AppceleratorTitanium","name":"AppceleratorTitanium"},{"key":"ArchLinuxPackages","name":"ArchLinuxPackages"},{"key":"Autotools","name":"Autotools"},{"key":"C","name":"C"},{"key":"C++","name":"C++"},{"key":"CFWheels","name":"CFWheels"},{"key":"CMake","name":"CMake"},{"key":"CUDA","name":"CUDA"},{"key":"CakePHP","name":"CakePHP"},{"key":"ChefCookbook","name":"ChefCookbook"},{"key":"Clojure","name":"Clojure"},{"key":"CodeIgniter","name":"CodeIgniter"},{"key":"CommonLisp","name":"CommonLisp"},{"key":"Composer","name":"Composer"},{"key":"Concrete5","name":"Concrete5"}]
seankhliao commented 2 years ago

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

ghost commented 2 years ago

How do you proof it not a bug. and i can 100% reproduce it.

ghost commented 2 years ago

发现问题比解决问题更难。为什么你一关了之？你知不知道为了构建最小可复现代码示例，我花了多少时间来研究？ 我认为这是一个bug，原因如下：

既然知道不对，为什么不让指出来？

ghost commented 2 years ago

The real expected looks very similar with fmt.Sprintf("%#v", var)

but with json format

{Status:"422 Unprocessable Entity", StatusCode:422, Proto:"HTTP/2.0", ProtoMajor:2, ProtoMinor:0, Header:http.Header{"Access-Control-Allow-Origin":[]string{"*"}, "Access-Control-Expose-Headers":[]string{"ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset"}, "Content-Length":[]string{"236"}, "Content-Security-Policy":[]string{"default-src 'none'"}, "Content-Type":[]string{"application/json; charset=utf-8"}, "Date":[]string{"Wed, 22 Dec 2021 09:01:35 GMT"}, "Referrer-Policy":[]string{"origin-when-cross-origin, strict-origin-when-cross-origin"}, "Server":[]string{"GitHub.com"}, "Strict-Transport-Security":[]string{"max-age=31536000; includeSubdomains; preload"}, "Vary":[]string{"Accept-Encoding, Accept, X-Requested-With"}, "X-Accepted-Oauth-Scopes":[]string{""}, "X-Content-Type-Options":[]string{"nosniff"}, "X-Frame-Options":[]string{"deny"}, "X-Github-Media-Type":[]string{"github.v3; format=json"}, "X-Github-Request-Id":[]string{"2270:3288:315B53:33D24D:61C2E96E"}, "X-Oauth-Scopes":[]string{"repo"}, "X-Ratelimit-Limit":[]string{"5000"}, "X-Ratelimit-Remaining":[]string{"4997"}, "X-Ratelimit-Reset":[]string{"1640167034"}, "X-Ratelimit-Resource":[]string{"core"}, "X-Ratelimit-Used":[]string{"3"}, "X-Xss-Protection":[]string{"0"}}, Body:http.http2transportResponseBody{cs:(*http.http2clientStream)(0xc000208300)}, ContentLength:236, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc0001ce000), TLS:(*tls.ConnectionState)(0xc0003aa0b0)}
orestonce commented 2 years ago

@cnmade json.Marshal都报错了，还扯后面的。 panic: json: unsupported type: func() (io.ReadCloser, error)

goroutine 1 [running]: main.main() make/main.go:24 +0x3ef

// Click here and start typing.
package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    hc, err := http.NewRequest("GET", "https://gitlab.com/api/v4/templates/gitignores", bytes.NewBuffer([]byte{}))
    if err != nil {
        panic(err)
    }
    clt := http.Client{}
    response, err := clt.Do(hc)
    if err != nil {
        panic(err)
    }
    jsonStr, err := json.Marshal(response)
    if err != nil {
        panic(err) // 这里都出错了
    }
    fmt.Printf("Response: '%+v' \n", string(jsonStr))
    fmt.Printf("Response raw: '%+v'\n", response)

    bodyAll, err := ioutil.ReadAll(response.Body)

    fmt.Println("response status: " + response.Status)
    fmt.Println("body " + string(bodyAll))
}
ghost commented 2 years ago

for any one who want to make json.Marshal really work, you may want to take: https://github.com/cnmade/goencode/blob/v1.17/json/encode.go

and the example work code like

package main

import (
    "bytes"
    "fmt"
    "github.com/cnmade/goencode/json"
    "io/ioutil"
    "net/http"
)

func main() {
    hc, err := http.NewRequest("GET", "https://gitlab.com/api/v4/templates/gitignores", bytes.NewBuffer([]byte{}))
    if err != nil {
        panic(err)
    }

    clt := http.Client{}
    response, err := clt.Do(hc)
    json.UnsupportedBehaviour = json.UnsupportedBehaviourWithNull
    jsonStr, err := json.Marshal(response)
    if err != nil {
        fmt.Printf("error: %#v\n", err.Error())
        panic(err)
    }
    fmt.Printf("Response: '%+v' \n", string(jsonStr))

    fmt.Printf("Response raw: '%+v'\n", response)

    bodyAll, err := ioutil.ReadAll(response.Body)

    fmt.Println("response status: " + response.Status)
    fmt.Println("body " + string(bodyAll))
}

After modify, json.Marshal will not throw Error, but encoding other valid Fields.

ghost commented 2 years ago

@orestonce 问题我知道，但go不处理，就离谱。 我提问题的意思是，这里有改进空间，可以让用户自己决定，是忽略解析不了的地方。 还是说直接报错，还是替代成其它固定的结构，或者是用fmt.Sprintf处理一下。 具体修改的代码见：


func unsupportedTypeEncoder(e *encodeState, v reflect.Value, _ encOpts) {
    if UnsupportedBehaviour != UnsupportedBehaviourWithError {
        switch UnsupportedBehaviour {
        case UnsupportedBehaviourWithSprintf:
            e.WriteString(fmt.Sprintf("'%+v'", v))
        case UnsupportedBehaviourWithNone:
            e.WriteString(UnsupportedBehaviourWithNone)
        case UnsupportedBehaviourWithNull:
            e.WriteString(UnsupportedBehaviourWithNull)
        default:
            e.error(&UnsupportedTypeError{v.Type()})
        }
        return
    }
    e.error(&UnsupportedTypeError{v.Type()})
}

你要相信一点，敢提问题，那肯定是有哪里不对，你能想到的，我也想到了。

orestonce commented 2 years ago

@cnmade 你的这种做法只是给json做了另一方面的改动，并无更好的优化效果。 我认为 json.Marshal报错就是json无法序列化对应的数据或者强行序列化后无法让另外的json解析器完全读出刚才传入的数据。所以不支持的类型直接报序列化错误我觉得是没毛病的。 至于fmt.Sprintf则没有地方需要反序列化，则可以填充处理即可。

orestonce commented 2 years ago

对于无法序列化/反序列化的字段，明确在代码里使用 json:- 告知json代码忽略该字段，是有必要的。

maodou1990 commented 2 years ago

@orestonce 问题我知道，但go不处理，就离谱。 我提问题的意思是，这里有改进空间，可以让用户自己决定，是忽略解析不了的地方。 还是说直接报错，还是替代成其它固定的结构，或者是用fmt.Sprintf处理一下。 具体修改的代码见：


func unsupportedTypeEncoder(e *encodeState, v reflect.Value, _ encOpts) {
  if UnsupportedBehaviour != UnsupportedBehaviourWithError {
      switch UnsupportedBehaviour {
      case UnsupportedBehaviourWithSprintf:
          e.WriteString(fmt.Sprintf("'%+v'", v))
      case UnsupportedBehaviourWithNone:
          e.WriteString(UnsupportedBehaviourWithNone)
      case UnsupportedBehaviourWithNull:
          e.WriteString(UnsupportedBehaviourWithNull)
      default:
          e.error(&UnsupportedTypeError{v.Type()})
      }
      return
  }
  e.error(&UnsupportedTypeError{v.Type()})
}

你要相信一点，敢提问题，那肯定是有哪里不对，你能想到的，我也想到了。

你所说的不是标准库应该做的，标准库要做的就是要返回错误码，由用户来决定是要怎么处理error，你所做的是要你自己去处理或者借助第三方库去处理，正如java所做的

LonelySally commented 2 years ago

对于无法序列化/反序列化的字段，明确在代码里使用 json:- 告知json代码忽略该字段，是有必要的。

本就该如此.response结构体内的Body的类型为io.ReadCloser本就不属于常规类型.譬如还有net.Conn bufio.Reader context.Context甚至于func() 等,这些非常规类型的导出就该使用json:-去屏蔽掉.我完全看不出楼主的槽点.=_=

LonelySally commented 2 years ago

@orestonce 问题我知道，但go不处理，就离谱。 我提问题的意思是，这里有改进空间，可以让用户自己决定，是忽略解析不了的地方。 还是说直接报错，还是替代成其它固定的结构，或者是用fmt.Sprintf处理一下。 具体修改的代码见：


func unsupportedTypeEncoder(e *encodeState, v reflect.Value, _ encOpts) {
    if UnsupportedBehaviour != UnsupportedBehaviourWithError {
        switch UnsupportedBehaviour {
        case UnsupportedBehaviourWithSprintf:
            e.WriteString(fmt.Sprintf("'%+v'", v))
        case UnsupportedBehaviourWithNone:
            e.WriteString(UnsupportedBehaviourWithNone)
        case UnsupportedBehaviourWithNull:
            e.WriteString(UnsupportedBehaviourWithNull)
        default:
            e.error(&UnsupportedTypeError{v.Type()})
        }
        return
    }
    e.error(&UnsupportedTypeError{v.Type()})
}

你要相信一点，敢提问题，那肯定是有哪里不对，你能想到的，我也想到了。

你所说的不是标准库应该做的，标准库要做的就是要返回错误码，由用户来决定是要怎么处理error，你所做的是要你自己去处理或者借助第三方库去处理，正如java所做的

完全可以自己在封装一层,通过反射区分出基础类型,剔除掉非基础类型的字段再做导出.