Copilot extension installation is extremely slow

[iamsubingyawali]

Describe the bug

I'm experiencing an issue while attempting to install the GitHub Copilot extension for the GitHub CLI on Linux. When running the following command:

gh ext install github/gh-copilot

The installation process appears to get stuck and doesn't complete. If I cancel the installation and try to reinstall the extension, the CLI reports that the extension already exists. However, when I try to run the gh copilot command, I receive the following error:

failed to run extension: fork/exec /home/user/.local/share/gh/extensions/gh-copilot/gh-copilot: no such file or directory

This suggests that the extension was not installed properly. I tried using gh ext browse and tried to install it from there as well but it too didn't work.

Environment

OS: Zorin OS 17.1 (Based on Ubuntu Jammy) GitHub CLI: v2.46.0 Shell: Bash v5.1.16

Expected Behavior

The GitHub Copilot extension should be installed correctly, and the gh copilot command should work as expected, providing access to the Copilot functionality in the CLI.

Please let me know if you need any additional information to investigate and resolve this issue.

[iamsubingyawali]

Opening this issue here: https://github.com/github/gh-copilot

[williammartin]

The installation process appears to get stuck and doesn't complete.

Just curious, how long did you wait? Reminds me of https://github.com/cli/cli/issues/8630 which would provide more visibility into this.

One thing you could do to help would be to provide the output with GH_DEBUG=api set, so that we can see the requests and responses being sent.

[iamsubingyawali]

I uninstalled and reinstalled the extension using command gh ext install github/gh-copilot --force with GH_DEBUG set to api and it took more than 40 minutes to get installed. Please find the logs in the attached text file: gh-copilot-installation-logs.txt

[williammartin]

That's interesting.

Here's the download request. It says this request took 40 minutes to complete which matches what you see. So either the network is slow (most likely) or writing the contents to the disk is slow.

* Request at 2024-03-23 14:49:54.682525976 +0545 +0545 m=+1.295778576
* Request to https://objects.githubusercontent.com/github-production-release-asset-2e65be/710311348/13242d70-70d9-46c2-bb8d-2717161bd09b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240323%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240323T090454Z&X-Amz-Expires=300&X-Amz-Signature=4add35aeea6db5060fd66be9d76488190cc73e9baf6d4f3a9503ef6ff3d8bb38&X-Amz-SignedHeaders=host&actor_id=45819206&key_id=0&repo_id=710311348&response-content-disposition=attachment%3B%20filename%3Dlinux-amd64&response-content-type=application%2Foctet-stream
> GET /github-production-release-asset-2e65be/710311348/13242d70-70d9-46c2-bb8d-2717161bd09b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240323%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240323T090454Z&X-Amz-Expires=300&X-Amz-Signature=4add35aeea6db5060fd66be9d76488190cc73e9baf6d4f3a9503ef6ff3d8bb38&X-Amz-SignedHeaders=host&actor_id=45819206&key_id=0&repo_id=710311348&response-content-disposition=attachment%3B%20filename%3Dlinux-amd64&response-content-type=application%2Foctet-stream 
> Host: objects.githubusercontent.com
> Accept: application/octet-stream
> Content-Type: application/json; charset=utf-8
> Referer: https://api.github.com/repos/github/gh-copilot/releases/assets/158140869
> Time-Zone: Asia/Kathmandu
> User-Agent: GitHub CLI 2.46.0
> X-Gh-Cache-Ttl: 30s

⣻
⣟< HTTP/2.0 200 OK
< Accept-Ranges: bytes
< Age: 1281
< Content-Disposition: attachment; filename=linux-amd64
< Content-Length: 31440896
< Content-Md5: KLgL5P2dNcXDbKX2hALcaA==
< Content-Type: application/octet-stream
< Date: Sat, 23 Mar 2024 09:04:55 GMT
< Etag: "0x8DC4AAC6A593CB2"
< Last-Modified: Fri, 22 Mar 2024 20:12:35 GMT
< Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< Via: 1.1 varnish, 1.1 varnish
< X-Cache: HIT, HIT
< X-Cache-Hits: 300, 0
< X-Ms-Blob-Type: BlockBlob
< X-Ms-Creation-Time: Fri, 22 Mar 2024 20:12:35 GMT
< X-Ms-Lease-State: available
< X-Ms-Lease-Status: unlocked
< X-Ms-Request-Id: a20a1da3-f01e-005c-1c95-7cb40d000000
< X-Ms-Server-Encrypted: true
< X-Ms-Version: 2020-10-02
< X-Served-By: cache-iad-kiad7000168-IAD, cache-dxb1470026-DXB
< X-Timer: S1711184696.716869,VS0,VE194

* Request took 40m14.964529305s

I'm curious, how long does it take to download the artifact from here in your browser? https://github.com/github/gh-copilot/releases/download/v1.0.1/linux-amd64

[iamsubingyawali]

Yes, it seemed quite interesting to me as well.

While other browsing/downloads seem to work at the expected speed, this specific artifact download is extremely slow, which is strange.

I'm not sure about the write speed on the disk, I noticed a significant difference in download speeds when I tried downloading this artifact and another file from testfile.org to the same location. The artifact downloaded much slower compared to the test file.

[williammartin]

I'm not sure about the write speed on the disk, I noticed a significant difference in download speeds when I tried downloading this artifact and another file from testfile.org to the same location. The artifact downloaded much slower compared to the test file.

Yeh I mean, it certainly sounds like something funny happening in the serving of this artifact (I just didn't want to rule out something on the client side). From your log it kind of looks like your TZ is Time-Zone: Asia/Kathmandu and this is being served from caches cache-iad-kiad7000168-IAD, cache-dxb1470026-DXB which would be Virginia and maybe Dubai (?), so maybe there's something funny happening there.

I might be able to track down someone on the platform side to look into this next week but it really seems like it is out of our hands in the CLI.

[iamsubingyawali]

I appreciate you looking into this and escalating.

I tried downloading the same file from a virtual machine in the South India region, and there were no issues. This clearly indicates an issue with either my network/ISP or GitHub's content delivery network, as you pointed out.

[williammartin]

Also, if you have the opportunity to try again, I'd be interested to see if it has resolved just by doing nothing (the best kind of resolution).

[iamsubingyawali]

I've tried it, but the issue still persists, or it has gotten even worse. Additionally, I've discovered that this problem extends to every artifact from GitHub, not just this particular one. It seems there's something really interesting happening between my network provider and GitHub's content delivery. I've gone ahead and created a support ticket with my provider to see if they can find out something.

[williammartin]

Ah that really sucks. Unfortunately this is going to be beyond what we in the CLI team can do to help. I'm going to close this now but if you find anything out or your network provider can point at something our end please drop a comment here and I can update whatever team might be able to look at it internally.

Good luck!

[williammartin]

Just a thought, I wonder if traceroute might show a hop that is particularly slow. Not sure though, definitely not my area of expertise!

[iamsubingyawali]

I just tried it with traceroute github.com and saw that the requests couldn't go further from 104.44.230.62 (9th hop including my local router). I waited for more than 40 mins but traceroute failed after requests being timed out for 255 max hops count. This hop IP seems to be MS datacenter IP in Singapore. I have no issues accessing github.com and performing pull/push to GitHub, so it again doesn't feel normal to my understanding.

[iamsubingyawali]

My network provider responded that they are aware of issues with GitHub and their network and that they are working to resolve them. So, I believe there is nothing I can do but wait.

[williammartin]

Thanks for the update, good to know that something could be pinpointed. Seems like best left between them and GitHub's network team.