search

prebid / prebid-server

Open-source solution for running real-time advertising auctions in the cloud.
https://prebid.org/product-suite/prebid-server/
Apache License 2.0
423 stars 718 forks source link

Data race in module on "StageBidderRequest" #3576

Open linux019 opened 5 months ago

linux019 commented 5 months ago

I need to access bidder aliases in the RequestWrapper to resolve adapter name (as modules on many stages receive bidder alias, not "bidder core name" = adapter name) race.log - full log race.tar.gz - test scripts & config

  1. Use pbs.yaml with enabled default module: 
    
external_url: "http://localhost:8000"

metrics: prometheus: port: 9100 auction_timeouts_ms: default: 60000 max: 600000

compression: response: enable_gzip: false request: enable_gzip: false

gdpr: enabled: true default_value: "0"

adapters: ix: endpoint: https://00000.lb.indexww.com/pbs disabled: false rubicon: disabled: false endpoint: "https://exapi-us-east.rubiconproject.com/a/api/exchange"

hooks: enabled: true modules: prebid: ortb2blocking: enabled: true

host_execution_plan: endpoints: "/openrtb2/auction": stages:

      bidder_request:
        groups:
          timeout: 100 #ms
          hook_sequence:
            - module_code: "prebid.ortb2blocking"
              hook_impl_code: "prebid.ortb2blocking"

2. Add following code here to read bidder aliases to the demo module 
https://github.com/prebid/prebid-server/blob/4870355f3ecfe7b32b275d5bf9ab7edbb991c51f/modules/prebid/ortb2blocking/hook_bidderrequest.go#L22
if reqExt, err := payload.Request.GetRequestExt(); err == nil {
    _ = reqExt.GetPrebid().Aliases
}
or apply patch with `git apply  file.patch` (in the project root directory)
4. run server with enabled race detector `go run -race main.go`
5. use "run.sh" from the archive to run 50 parallel requests (it needs to run several times the shell script with curl to get a data race)
6. result

================== WARNING: DATA RACE Write at 0x00c000342c08 by goroutine 703: github.com/prebid/prebid-server/v2/openrtb_ext.(RequestWrapper).GetRequestExt() github.com/prebid/prebid-server/v2/openrtb_ext/request_wrapper.go:127 +0xa4 github.com/prebid/prebid-server/v2/modules/prebid/ortb2blocking.Module.HandleBidderRequestHook() github.com/prebid/prebid-server/v2/modules/prebid/ortb2blocking/module.go:25 +0x85 github.com/prebid/prebid-server/v2/modules/prebid/ortb2blocking.(Module).HandleBidderRequestHook()

:1 +0x16d github.com/prebid/prebid-server/v2/hooks/hookexecution.(*hookExecutor).ExecuteBidderRequestStage.func1() github.com/prebid/prebid-server/v2/hooks/hookexecution/executor.go:199 +0x161 github.com/prebid/prebid-server/v2/hooks/hookexecution.executeHook[go.shape.interface { HandleBidderRequestHook(context.Context, github.com/prebid/prebid-server/v2/hooks/hookstage.ModuleInvocationContext, github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload) (github.com/prebid/prebid-server/v2/hooks/hookstage.HookResult[github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload], error) },go.shape.struct { Request *github.com/prebid/prebid-server/v2/openrtb_ext.RequestWrapper; Bidder string }].func1() github.com/prebid/prebid-server/v2/hooks/hookexecution/execution.go:108 +0x1e6 Previous read at 0x00c000342c08 by goroutine 576: github.com/prebid/prebid-server/v2/openrtb_ext.(*RequestWrapper).rebuildRequestExt() github.com/prebid/prebid-server/v2/openrtb_ext/request_wrapper.go:285 +0x3c github.com/prebid/prebid-server/v2/openrtb_ext.(*RequestWrapper).RebuildRequest() github.com/prebid/prebid-server/v2/openrtb_ext/request_wrapper.go:203 +0x9a github.com/prebid/prebid-server/v2/exchange.(*bidderAdapter).requestBid() github.com/prebid/prebid-server/v2/exchange/bidder.go:150 +0x24b github.com/prebid/prebid-server/v2/exchange.(*validatedBidder).requestBid() github.com/prebid/prebid-server/v2/exchange/bidder_validate_bids.go:35 +0x216 github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.func1() github.com/prebid/prebid-server/v2/exchange/exchange.go:720 +0x98b github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.(*exchange).recoverSafely.func2() github.com/prebid/prebid-server/v2/exchange/exchange.go:829 +0x1c4 github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.gowrap1() github.com/prebid/prebid-server/v2/exchange/exchange.go:751 +0x104 Goroutine 703 (running) created at: github.com/prebid/prebid-server/v2/hooks/hookexecution.executeHook[go.shape.interface { HandleBidderRequestHook(context.Context, github.com/prebid/prebid-server/v2/hooks/hookstage.ModuleInvocationContext, github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload) (github.com/prebid/prebid-server/v2/hooks/hookstage.HookResult[github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload], error) },go.shape.struct { Request *github.com/prebid/prebid-server/v2/openrtb_ext.RequestWrapper; Bidder string }]() github.com/prebid/prebid-server/v2/hooks/hookexecution/execution.go:105 +0x390 github.com/prebid/prebid-server/v2/hooks/hookexecution.executeGroup[go.shape.interface { HandleBidderRequestHook(context.Context, github.com/prebid/prebid-server/v2/hooks/hookstage.ModuleInvocationContext, github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload) (github.com/prebid/prebid-server/v2/hooks/hookstage.HookResult[github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload], error) },go.shape.struct { Request *github.com/prebid/prebid-server/v2/openrtb_ext.RequestWrapper; Bidder string }].func1() github.com/prebid/prebid-server/v2/hooks/hookexecution/execution.go:78 +0x22c github.com/prebid/prebid-server/v2/hooks/hookexecution.executeGroup[go.shape.interface { HandleBidderRequestHook(context.Context, github.com/prebid/prebid-server/v2/hooks/hookstage.ModuleInvocationContext, github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload) (github.com/prebid/prebid-server/v2/hooks/hookstage.HookResult[github.com/prebid/prebid-server/v2/hooks/hookstage.BidderRequestPayload], error) },go.shape.struct { Request *github.com/prebid/prebid-server/v2/openrtb_ext.RequestWrapper; Bidder string }].gowrap1() github.com/prebid/prebid-server/v2/hooks/hookexecution/execution.go:79 +0xf8 Goroutine 576 (running) created at: github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids() github.com/prebid/prebid-server/v2/exchange/exchange.go:751 +0x18f github.com/prebid/prebid-server/v2/exchange.(*exchange).HoldAuction() github.com/prebid/prebid-server/v2/exchange/exchange.go:378 +0x2624 github.com/prebid/prebid-server/v2/endpoints/openrtb2.(*endpointDeps).Auction() github.com/prebid/prebid-server/v2/endpoints/openrtb2/auction.go:264 +0x1adb github.com/prebid/prebid-server/v2/endpoints/openrtb2.(*endpointDeps).Auction-fm() :1 +0x73 github.com/julienschmidt/httprouter.(*Router).ServeHTTP() github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0xf0b github.com/prebid/prebid-server/v2/router.(*Router).ServeHTTP() :1 +0x64 github.com/prebid/prebid-server/v2/router.SupportCORS.(*Cors).Handler.func2() github.com/rs/cors@v1.8.2/cors.go:231 +0x2f1 net/http.HandlerFunc.ServeHTTP() net/http/server.go:2166 +0x47 github.com/prebid/prebid-server/v2/router.NoCache.ServeHTTP() github.com/prebid/prebid-server/v2/router/router.go:119 +0x421 github.com/prebid/prebid-server/v2/router.(*NoCache).ServeHTTP() :1 +0x69 net/http.serverHandler.ServeHTTP() net/http/server.go:3137 +0x2a1 net/http.(*conn).serve() net/http/server.go:2039 +0x13c4 net/http.(*Server).Serve.gowrap3() net/http/server.go:3285 +0x4f ================== ================== WARNING: DATA RACE Read at 0x00c00219dde0 by goroutine 841: github.com/prebid/prebid-server/v2/openrtb_ext.(*RequestWrapper).GetRequestExt() github.com/prebid/prebid-server/v2/openrtb_ext/request_wrapper.go:128 +0x10a github.com/prebid/prebid-server/v2/modules/prebid/ortb2blocking.Module.HandleBidderRequestHook() ```
linux019 commented 5 months ago

Moreover just running tests with enabled race detector go test -race ./... produces a lot of warnings in "exchange"

WARNING: DATA RACE
Read at 0x00c00053ae48 by goroutine 647:
  net/http.Header.Clone()
      /usr/lib/go/src/net/http/header.go:101 +0x5697
  github.com/prebid/prebid-server/v2/exchange.(*bidderAdapter).requestBid()
      exchange/bidder.go:174 +0x6b8
  github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.func1()
      exchange/exchange.go:733 +0x98b
  github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.(*exchange).recoverSafely.func2()
      exchange/exchange.go:842 +0x1c4
  github.com/prebid/prebid-server/v2/exchange.(*exchange).getAllBids.gowrap1()
      exchange/exchange.go:764 +0x104

Previous write at 0x00c00053ae48 by goroutine 648:
  net/http.Header.Clone()
      /usr/lib/go/src/net/http/header.go:114 +0x58d4
  github.com/prebid/prebid-server/v2/exchange.(*bidderAdapter).requestBid()
      exchange/bidder.go:174 +0x6b8

And all OpenRTB structures in https://github.com/prebid/prebid-server/blob/master/openrtb_ext/request_wrapper.go have accessors which modifies internal data on read access and this request wrapper is shared between adapters or hooks which are spawned in goroutines

SyntaxNode commented 5 months ago

Closed accidently during backlog grooming. Reopened.