search

projectdiscovery / httpx-action

HTTP Web Server probing
MIT License
7 stars 8 forks source link

403 Forbidden Errors when downloading dependencies #6

Closed rxerium closed 10 months ago

rxerium commented 11 months ago

Hi there,

I am using httpX action in my workflow and as it is downloading the dependencies I receive the following error:

go: downloading github.com/lucasb-eyer/go-colorful v1.2.0
go: downloading github.com/rivo/uniseg v0.4.4
go: downloading github.com/tidwall/pretty v1.2.1
go: downloading github.com/dlclark/regexp2 v1.8.1
go: downloading github.com/tidwall/tinyqueue v0.1.1
Error: ../../../../go/pkg/mod/github.com/mholt/archiver@v3.1.1+incompatible/lz4.go:8:2: github.com/pierrec/lz4@v2.6.1+incompatible: reading https://proxy.golang.org/github.com/pierrec/lz4/@v/v2.6.1+incompatible.zip: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.2/u_handshake_client.go:15:2: github.com/klauspost/compress@v1.16.7: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.16.7.zip: 403 Forbidden
Error: Process completed with exit code 1.

In this particular example my input list contains 1000 hosts however I've received this error for other scans too. Below is my config however I don't think this will help as its failing even before the scan starts:

      - name: httpX Scan
        uses: projectdiscovery/httpx-action@main
        with:
          list: .github/input/input.txt
          output: ~/output.txt
          flags: "-config .github/configurations/httpx/httpx-config.yaml"

Any ideas why this is the case?

Many thanks

Rishi

rxerium commented 11 months ago

thanks all!

rxerium commented 10 months ago

Good afternoon!

I am still experiencing this error unfortunately, here is the updated log:

Run projectdiscovery/httpx-action@main
Run [ ! -x /home/runner/go/bin/httpx ] && go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
go: downloading github.com/projectdiscovery/httpx v1.3.5
go: downloading github.com/projectdiscovery/gologger v1.1.11
go: downloading github.com/projectdiscovery/utils v0.0.54
go: downloading github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057
go: downloading github.com/PuerkitoBio/goquery v1.8.1
go: downloading github.com/go-faker/faker/v4 v4.1.1
go: downloading github.com/go-rod/rod v0.114.0
go: downloading github.com/logrusorgru/aurora v2.0.3+incompatible
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/projectdiscovery/asnmap v1.0.4
go: downloading github.com/projectdiscovery/cdncheck v1.0.9
go: downloading github.com/projectdiscovery/clistats v0.0.19
go: downloading github.com/projectdiscovery/dsl v0.0.21
go: downloading github.com/projectdiscovery/fastdialer v0.0.37
go: downloading github.com/projectdiscovery/fdmax v0.0.4
go: downloading github.com/projectdiscovery/goconfig v0.0.1
go: downloading github.com/projectdiscovery/goflags v0.1.20
go: downloading github.com/projectdiscovery/hmap v0.0.16
go: downloading github.com/projectdiscovery/mapcidr v1.1.2
go: downloading github.com/projectdiscovery/ratelimit v0.0.9
go: downloading github.com/projectdiscovery/rawhttp v0.1.18
go: downloading github.com/projectdiscovery/retryablehttp-go v1.0.26
go: downloading github.com/projectdiscovery/tlsx v1.1.1
go: downloading github.com/projectdiscovery/wappalyzergo v0.0.109
go: downloading github.com/remeh/sizedwaitgroup v1.0.0
go: downloading golang.org/x/exp v0.0.0-20230810033253-352e893a4cad
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/mholt/archiver v3.1.1+incompatible
go: downloading gopkg.in/djherbis/times.v1 v1.3.0
go: downloading github.com/andybalholm/cascadia v1.3.1
go: downloading golang.org/x/net v0.14.0
go: downloading golang.org/x/text v0.12.0
go: downloading github.com/ysmood/goob v0.4.0
go: downloading github.com/ysmood/got v0.34.1
go: downloading github.com/ysmood/gson v0.7.3
go: downloading github.com/ysmood/fetchup v0.2.3
go: downloading github.com/ysmood/leakless v0.8.0
go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2
go: downloading github.com/projectdiscovery/retryabledns v1.0.35
go: downloading github.com/weppos/publicsuffix-go v0.30.1-0.20230422193905-8fecedd899db
go: downloading github.com/yl2chen/cidranger v1.0.2
go: downloading github.com/projectdiscovery/freeport v0.0.5
go: downloading github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
go: downloading github.com/hashicorp/go-version v1.6.0
go: downloading github.com/kataras/jwt v0.1.8
go: downloading github.com/projectdiscovery/gostruct v0.0.1
go: downloading github.com/spaolacci/murmur3 v1.1.0
go: downloading github.com/dimchansky/utfbom v1.1.1
go: downloading github.com/projectdiscovery/networkpolicy v0.0.6
go: downloading github.com/refraction-networking/utls v1.5.2
go: downloading github.com/ulule/deepcopier v0.0.0-20200430083143-45decc6639b6
go: downloading github.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/jaytaylor/html2text v0.0.0-20230321000545-74c2419ad056
go: downloading github.com/mfonda/simhash v0.0.0-20151007195837-79f94a1100d6
go: downloading github.com/hdm/jarm-go v0.0.7
go: downloading go.uber.org/multierr v1.11.0
go: downloading github.com/corpix/uarand v0.2.0
go: downloading github.com/hbakhtiyor/strsim v0.0.0-20190107154042-4d2bbb273edf
go: downloading github.com/microcosm-cc/bluemonday v1.0.25
go: downloading github.com/rs/xid v1.5.0
go: downloading github.com/projectdiscovery/blackrock v0.0.1
go: downloading github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809
go: downloading github.com/cloudflare/cfssl v1.6.4
go: downloading github.com/miekg/dns v1.1.55
go: downloading github.com/shirou/gopsutil/v3 v3.23.7
go: downloading github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d
go: downloading github.com/Masterminds/semver/v3 v3.2.1
go: downloading github.com/charmbracelet/glamour v0.6.0
go: downloading github.com/cheggaaa/pb/v3 v3.1.4
go: downloading github.com/google/go-github/v30 v30.1.0
go: downloading github.com/minio/selfupdate v0.6.0
go: downloading golang.org/x/oauth2 v0.11.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012[64](https://github.com/kyndio/nuclei-automation/actions/runs/6614288467/job/17963945560#step:7:65)4-bacd9c7ef1dd
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/dsnet/compress v0.0.1
go: downloading github.com/golang/snappy v0.0.4
go: downloading github.com/nwaples/rardecode v1.1.3
go: downloading github.com/pierrec/lz4 v2.6.1+incompatible
go: downloading github.com/ulikunitz/xz v0.5.11
go: downloading github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8
go: downloading github.com/sashabaranov/go-openai v1.14.2
go: downloading github.com/andybalholm/brotli v1.0.5
go: downloading github.com/cloudflare/circl v1.3.3
go: downloading github.com/gaukas/godicttls v0.0.4
go: downloading github.com/klauspost/compress v1.16.7
go: downloading github.com/quic-go/quic-go v0.37.4
go: downloading golang.org/x/crypto v0.12.0
go: downloading golang.org/x/sys v0.11.0
go: downloading github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248
go: downloading github.com/akrylysov/pogreb v0.10.1
go: downloading github.com/syndtr/goleveldb v1.0.0
go: downloading github.com/tidwall/buntdb v1.3.0
go: downloading go.etcd.io/bbolt v1.3.7
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/ssor/bom v0.0.0-2017071[81](https://github.com/kyndio/nuclei-automation/actions/runs/6614288467/job/17963945560#step:7:83)23548-63[86](https://github.com/kyndio/nuclei-automation/actions/runs/6614288467/job/17963945560#step:7:88)211fdfcf
go: downloading github.com/kljensen/snowball v0.8.0
go: downloading github.com/aymerick/douceur v0.2.0
go: downloading github.com/tklauser/go-sysconf v0.3.11
go: downloading github.com/muesli/termenv v0.15.1
go: downloading github.com/yuin/goldmark v1.5.4
go: downloading github.com/yuin/goldmark-emoji v1.0.1
go: downloading github.com/VividCortex/ewma v1.2.0
go: downloading github.com/fatih/color v1.15.0
go: downloading github.com/mattn/go-colorable v0.1.13
go: downloading github.com/mattn/go-isatty v0.0.19
go: downloading github.com/mattn/go-runewidth v0.0.14
go: downloading github.com/google/go-querystring v1.1.0
go: downloading aead.dev/minisign v0.2.0
go: downloading github.com/tidwall/btree v1.6.0
go: downloading github.com/tidwall/gjson v1.14.4
go: downloading github.com/tidwall/grect v0.1.4
go: downloading github.com/tidwall/match v1.1.1
go: downloading github.com/tidwall/rtred v0.1.2
go: downloading github.com/gorilla/css v1.0.0
go: downloading github.com/google/certificate-transparency-go v1.1.4
go: downloading github.com/tklauser/numcpus v0.6.0
go: downloading github.com/alecthomas/chroma v0.10.0
go: downloading github.com/muesli/reflow v0.3.0
go: downloading github.com/aymanbagabas/go-osc52/v2 v2.0.1
go: downloading github.com/lucasb-eyer/go-colorful v1.2.0
go: downloading github.com/rivo/uniseg v0.4.4
go: downloading github.com/tidwall/pretty v1.2.1
go: downloading github.com/dlclark/regexp2 v1.8.1
go: downloading github.com/tidwall/tinyqueue v0.1.1
Error: ../../../../go/pkg/mod/github.com/mholt/archiver@v3.1.1+incompatible/lz4.go:8:2: github.com/pierrec/lz4@v2.6.1+incompatible: reading https://proxy.golang.org/github.com/pierrec/lz4/@v/v2.6.1+incompatible.zip: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.2/u_handshake_client.go:15:2: github.com/klauspost/compress@v1.16.7: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.16.7.zip: 403 Forbidden
Error: Process completed with exit code 1.

Many thanks,

Rishi

dogancanbakir commented 10 months ago

@rxerium, I'll take another look at this, but as a possible workaround, you can try installing the following packages before using httpx-action.

GOPROXY=direct go get github.com/pierrec/lz4@v2.6.1
GOPROXY=direct go get github.com/klauspost/compress@v1.16.7
rxerium commented 10 months ago

Thanks @dogancanbakir I've ran the following step in my workflow however this errors:

      - name: httpX requirements
        run: |
          go get github.com/pierrec/lz4@v2.6.1
          go get github.com/klauspost/compress@v1.16.7

Error:

Run go get github.com/pierrec/lz4@v[2](https://github.com/rxerium/nuclei-action/actions/runs/6641303499/job/18043506916#step:5:2).6.1
  go get github.com/pierrec/lz4@v2.6.1
  go get github.com/klauspost/compress@v1.16.7
  shell: /usr/bin/bash -e {0}
go: go.mod file not found in current directory or any parent directory.
    'go get' is no longer supported outside a module.
    To build and install a command, use 'go install' with a version,
    like 'go install example.com/cmd@latest'
    For more information, see https://golang.org/doc/go-get-install-deprecation
    or run 'go help get' or 'go help install'.
Error: Process completed with exit code 1.
dogancanbakir commented 10 months ago

Of course, what was I thinking! Give me some time please, I'll look into this.

dogancanbakir commented 10 months ago

@rxerium, I just made a small change. Can you retry and let me know how it goes? This is the branch https://github.com/projectdiscovery/httpx-action/tree/goproxy_direct

      - name: httpX Scan
        uses: projectdiscovery/httpx-action@goproxy_direct
        with:
          list: .github/input/input.txt
          output: ~/output.txt
          flags: "-config .github/configurations/httpx/httpx-config.yaml"
rxerium commented 10 months ago

I've ran a few tests and I've not encountered the issue!! Thank you so much for the quick turnaround 🙌

dogancanbakir commented 10 months ago

Closing this. As a note, to avoid this issue in the future, add 'GOPROXY: direct' in your env vars.

rxerium commented 10 months ago

Will do! Thanks again :)

rxerium commented 10 months ago

Hey @dogancanbakir, will running the below step (before httpX) suffice:

      - name: httpX GOPROXY env vars
        run: |
          export GOPROXY=direct
dogancanbakir commented 10 months ago

@rxerium, Yes, hopefully 😁

rxerium commented 10 months ago

Unfortunately this didn't work

go: downloading github.com/klauspost/compress v1.16.7
Error: ../../../../go/pkg/mod/github.com/mholt/archiver@v3.1.1+incompatible/lz4.go:8:2: github.com/pierrec/lz4@v2.6.1+incompatible: reading https://proxy.golang.org/github.com/pierrec/lz4/@v/v2.6.1+incompatible.zip: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.3/u_handshake_client.go:15:2: github.com/klauspost/compress@v1.16.7: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.16.7.zip: 403 Forbidden
Error: Process completed with exit code 1.

I'll use the projectdiscovery/httpx-action@goproxy_direct branch in the interim.

rxerium commented 10 months ago

Just picked up another error FYI:

go: downloading github.com/refraction-networking/utls v1.5.3
go: downloading github.com/miekg/dns v1.1.56
go: downloading github.com/denisbrodbeck/machineid v1.0.1
go: downloading github.com/pierrec/lz4 v2.6.1+incompatible
go: downloading github.com/klauspost/compress v1.16.7
go: downloading golang.org/x/crypto v0.14.0
go: downloading golang.org/x/sys v0.13.0
Error: ../../../../go/pkg/mod/github.com/andybalholm/cascadia@v1.3.1/pseudo_classes.go:9:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/andybalholm/cascadia@v1.3.1/pseudo_classes.go:10:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/go-faker/faker/v4@v4.1.1/lorem.go:7:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/go-faker/faker/v4@v4.1.1/lorem.go:8:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/miekg/dns@v1.1.56/udp.go:9:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/miekg/dns@v1.1.56/udp.go:10:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/miekg/dns@v1.1.56/listen_reuseport.go:11:2: unrecognized import path "golang.org/x/sys": reading https://golang.org/x/sys?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/asnmap@v1.0.5/libs/client.go:19:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/weppos/publicsuffix-go@v0.30.1-0.20230422193905-8fecedd899db/publicsuffix/publicsuffix.go:16:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/cloudflare/circl@v1.3.3/math/fp25519/fp_amd64.go:7:2: unrecognized import path "golang.org/x/sys": reading https://golang.org/x/sys?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/cloudflare/circl@v1.3.3/xof/xof.go:13:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/cloudflare/circl@v1.3.3/xof/xof.go:14:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.3/internal/helper/typeconv.go:6:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.3/cipher_suites.go:24:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.3/key_schedule.go:16:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/refraction-networking/utls@v1.5.3/u_prng.go:23:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/zmap/zcrypto@v0.0.0-20230422215203-9a665e1e9968/x[50](https://github.com/kyndio/nuclei-automation/actions/runs/6717965823/job/18256776656#step:8:52)9/x509.go:43:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/retryablehttp-go@v1.0.31/client.go:8:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/google/certificate-transparency-go@v1.1.4/x509/x509.go:73:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/cloudflare/cfssl@v1.6.4/helpers/helpers.go:30:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/cloudflare/cfssl@v1.6.4/helpers/helpers.go:31:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/httpx@v1.3.6/common/httpx/httpx.go:26:2: unrecognized import path "golang.org/x/net": reading https://golang.org/x/net?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/httpx@v1.3.6/common/httpx/encodings.go:10:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/httpx@v1.3.6/common/httpx/encodings.go:11:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/httpx@v1.3.6/common/httpx/encodings.go:12:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/projectdiscovery/httpx@v1.3.6/common/httpx/encodings.go:13:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/mfonda/simhash@v0.0.0-201[51](https://github.com/kyndio/nuclei-automation/actions/runs/6717965823/job/18256776656#step:8:53)007195837-79f94a1100d6/simhash.go:15:2: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/github.com/google/go-github/v30@v30.1.0/github/git_commits.go:16:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: ../../../../go/pkg/mod/aead.dev/minisign@v0.2.0/private.go:22:2: unrecognized import path "golang.org/x/crypto": reading https://golang.org/x/crypto?go-get=1: 403 Forbidden
Error: Process completed with exit code 1.