Closed NobodyXu closed 3 months ago
We only use GITHUB_TOKEN to speedup the access, and we actually have a few requests created without the token.
adding debug info
GITHUB_TOKEN=ghp_XXXXX cargo binstall --log-level debug --strategies crate-meta-data --git https://github.com/TransitApp/gtfsRtProcessor.git gtfsrt-analyzer
DEBUG using (/Users/scullionw/.cargo) as cargo home
DEBUG Using install path: /Users/scullionw/.cargo/bin
DEBUG get_target_from_rustc()=Some("aarch64-apple-darwin")
INFO resolve: Resolving package: 'gtfsrt-analyzer'
DEBUG Shallow cloning https://github.com/TransitApp/gtfsRtProcessor.git to /var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9 with worktree
Username for https://github.com: scullionw
Password for https://scullionw@github.com:
INFO Cloning with worktree::receiving pack: Enumerating objects → 682.0 objects
INFO Cloning with worktree::receiving pack: Compressing objects → 10.0/470.0 objects [2%]
INFO Cloning with worktree::receiving pack::read pack → 9.2MB
INFO ℹCloning with worktree::receiving pack::create index file::indexing → done 682.0 objects in 0.34s (2.0K objects/s)
INFO ℹCloning with worktree::receiving pack::create index file::decompressing → done 19.9MB in 0.34s (57.8MB/s)
INFO ℹCloning with worktree::receiving pack::create index file::Resolving → done 682.0 objects in 0.11s (6.2K objects/s)
INFO ℹCloning with worktree::receiving pack::create index file::Decoding → done 21.9MB in 0.11s (199.1MB/s)
INFO ✓Cloning with worktree::receiving pack::create index file::writing index file → done 20.2KB in 0.00s (121.9MB/s)
INFO ✓Cloning with worktree::receiving pack::create index file → done 682.0 objects in 0.46s (1.5K objects/s)
INFO ℹCloning with worktree::receiving pack::read pack → done 14.1MB in 0.75s (18.8MB/s)
INFO ℹCloning with worktree::receiving pack::checkout → done 217.0 files in 0.03s (7.8K files/s)
INFO ℹCloning with worktree::receiving pack::writing → done 20.8MB in 0.03s (746.2MB/s)
DEBUG load_manifest_from_workspace_inner{workspace_path="/var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9" crate_name="gtfsrt-analyzer"}: Loading manifest of crate gtfsrt-analyzer from workspace: /var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9
DEBUG load_manifest_from_workspace_inner{workspace_path="/var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9" crate_name="gtfsrt-analyzer"}: Loading from /var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9/Cargo.toml, manifest.package.name = None
DEBUG load_manifest_from_workspace_inner{workspace_path="/var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9" crate_name="gtfsrt-analyzer"}: Loading from /var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9/apps/processor/Cargo.toml, manifest.package.name = Some(
"processor",
)
DEBUG load_manifest_from_workspace_inner{workspace_path="/var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9" crate_name="gtfsrt-analyzer"}: Loading from /var/folders/rf/j5ddky1d5m1g38339t6yt66h0000gn/T/.tmpUOM4w9/apps/gtfsrt-analyzer/Cargo.toml, manifest.package.name = Some(
"gtfsrt-analyzer",
)
DEBUG resolve: Building metadata for target: aarch64-apple-darwin
DEBUG resolve: Found metadata: PkgMeta { pkg_url: None, pkg_fmt: None, bin_dir: None, signing: None, overrides: {} }
DEBUG resolve: Building metadata for target: universal-apple-darwin
DEBUG resolve: Found metadata: PkgMeta { pkg_url: None, pkg_fmt: None, bin_dir: None, signing: None, overrides: {} }
DEBUG resolve: Building metadata for target: universal2-apple-darwin
DEBUG resolve: Found metadata: PkgMeta { pkg_url: None, pkg_fmt: None, bin_dir: None, signing: None, overrides: {} }
DEBUG resolve: Building metadata for target: x86_64-apple-darwin
DEBUG resolve: Found metadata: PkgMeta { pkg_url: None, pkg_fmt: None, bin_dir: None, signing: None, overrides: {} }
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}:do_send_request{self=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: {}, poisoned: false, .. } } }) request=Request { method: HEAD, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("github.com")), port: None, path: "/TransitApp/gtfsRtProcessor", query: None, fragment: None }, headers: {} } url=Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("github.com")), port: None, path: "/TransitApp/gtfsRtProcessor", query: None, fragment: None }}: Using system DNS resolver configuration
INFO get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: HEAD on https://github.com/TransitApp/gtfsRtProcessor is not allowed, fallback to GET
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
INFO get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: HEAD on https://github.com/TransitApp/gtfsRtProcessor is not allowed, fallback to GET
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
WARN resolve: Error while checking fetcher invalid url: Failed to download from remote: could not GET https://github.com/TransitApp/gtfsRtProcessor: HTTP status client error (404 Not Found) for url (https://github.com/TransitApp/gtfsRtProcessor)
INFO get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: HEAD on https://github.com/TransitApp/gtfsRtProcessor is not allowed, fallback to GET
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, num_request: 1, per: 10ms, until: Instant { tv_sec: 1002538, tv_nsec: 841163333 }, state: Ready { rem: 1 } }, poisoned: false, .. }, hosts_to_delay: Mutex { data: <locked>, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
WARN resolve: Error while checking fetcher invalid url: Failed to download from remote: could not GET https://github.com/TransitApp/gtfsRtProcessor: HTTP status client error (404 Not Found) for url (https://github.com/TransitApp/gtfsRtProcessor)
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: {}, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
INFO get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: {}, poisoned: false, .. } } })}: HEAD on https://github.com/TransitApp/gtfsRtProcessor is not allowed, fallback to GET
DEBUG get_repo_info{self=Data { name: "gtfsrt-analyzer", version: "1.2.1", repo: Some("https://github.com/TransitApp/gtfsRtProcessor"), repo_info: OnceCell { value: None } } client=Client(Inner { client: Client { accepts: Accepts { gzip: true, brotli: true, deflate: true }, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "cargo-binstall/1.6.4"} }, service: DelayRequest { inner: Mutex { data: <locked>, poisoned: false, .. }, hosts_to_delay: Mutex { data: {}, poisoned: false, .. } } })}: Downloading from: 'https://github.com/TransitApp/gtfsRtProcessor'
WARN resolve: Error while checking fetcher invalid url: Failed to download from remote: could not GET https://github.com/TransitApp/gtfsRtProcessor: HTTP status client error (404 Not Found) for url (https://github.com/TransitApp/gtfsRtProcessor)
WARN resolve: Error while checking fetcher invalid url: Failed to download from remote: could not GET https://github.com/TransitApp/gtfsRtProcessor: HTTP status client error (404 Not Found) for url (https://github.com/TransitApp/gtfsRtProcessor)
DEBUG run time: 9.839548041s
ERROR Fatal error:
× For crate gtfsrt-analyzer: Fallback to cargo-install is disabled
╰─▶ Fallback to cargo-install is disabled
and ive tried with and without this in the Cargo.toml
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/gtfsrt-analyzer-v{ version }/{ name }-v{ version }-{ target }.tar.gz"
bin-dir = "{ bin }{ binary-ext }"
pkg-fmt = "tgz"
this is the cargo toml
[package]
name = "gtfsrt-analyzer"
version = "1.2.1"
repository = "https://github.com/TransitApp/gtfsRtProcessor"
edition.workspace = true
authors.workspace = true
[dependencies]
bgtfs.workspace = true
gtfsrt.workspace = true
tapis.workspace = true
chrono.workspace = true
eyre.workspace = true
url.workspace = true
serde.workspace = true
serde_json.workspace = true
reqwest.workspace = true
clap = { version = "4.3.19", features = ["derive"] }
processor = { path = "../processor" }
trips_matcher = { path = "../../crates/trips-matcher" }
predictions = { path = "../../crates/predictions" }
reports = { path = "../../crates/reports" }
gtfs_rt_redis = { path = "../../crates/gtfs-rt-redis" }
gtfs_rt = { path = "../../crates/gtfs-rt" }
recipe_assignment = { path = "../../crates/recipe-assignment" }
utils = { path = "../../crates/utils" }
inquire = "0.6.2"
color-eyre = "0.6"
dialoguer = "0.11.0"
thiserror = "1.0.56"
colored = "2.1.0"
chrono-tz = "0.8.6"
toml = "0.8.8"
directories = "5"
confique = "0.2.3"
# Metadata for cargo-binstall
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/gtfsrt-analyzer-v{ version }/{ name }-v{ version }-{ target }.tar.gz"
bin-dir = "{ bin }{ binary-ext }"
pkg-fmt = "tgz"
Thanks, just need some help: can you curl
the private repository, if you provided the GitHub token as bearer token authentication?
If that works, then I could fix it by simply adding it to all repository access.
maybe im not doing it right, but
curl --head \
--url "https://github.com/TransitApp/gtfsRtProcessor" \
--header "Authorization: Bearer ghp_X"
is still giving me a 404 (with the actual token)
what does seem to work is
curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer gh_XXXXX" \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/TransitApp/gtfsRtProcessor/releases
then pulling the asset id from that, and then calling
curl -L \
-H "Accept: application/octet-stream" \
-H "Authorization: Bearer ghp_XXX" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-o mytar.tar.gz \
https://api.github.com/repos/TransitApp/gtfsRtProcessor/releases/assets/165152044
docs: https://docs.github.com/en/rest/releases/assets?apiVersion=2022-11-28#get-a-release-asset
Thanks I see, it would require some more modifications to cargo-binstall then.
NOTE to myself, cargo-binstall would also need https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#get-a-repository
For listing repository:
r#"
query {{
repository(owner:"{owner}",name:"{repo}") {{
owner {{
login
}}
name
private: isPrivate
}}
}}"#
Example output:
{
"data": {
"repository": {
"owner": {
"login": "cargo-bins"
},
"name": "cargo-binstall",
"private": false
}
}
}
@scullionw Released cargo-binstall 1.7.0
we are currently facing an issue, and not sure if we are doing something wrong
it is a private github repo, and the crate is not in any registry. the release artifact is stored in the github releases.
it seems the github token works for pulling the Cargo.toml from the github repo, but then fails for some other requests for the same url? The github token has all privileges
Is --git the correct mode to use?
Originally posted by @scullionw in https://github.com/cargo-bins/cargo-binstall/discussions/1535#discussioncomment-9299419