Sidekiq-ent/pro no longer downloading

[J-Swift]

Hello again. Something weird that suddenly started occurring and I'm not sure where the issue lies but wanted to get a convo going in case you have any ideas.

We are on sidekiq enterprise, which requires auth info for bundler to download the gems. This was working fine for ~3 weeks, then suddenly on monday I can no longer rebuild the environment... I've reinstalled nix several times, I've upgraded to seqouia, I've updated from ruby 3.2 to 3.3... still the same issue during gem fetching.

$ nix log /nix/store/78q9dgkwym0hgp2k7a5xrw7fwrx5zqd4-sidekiq-ent-2.5.3.gem.drv
warning: The interpretation of store paths arguments ending in `.drv` recently changed. If this command is now failing try again with '/nix/store/78q9dgkwym0hgp2k7a5xrw7fwrx5zqd4-sidekiq-ent-2.5.3.gem.drv^*'

trying https://enterprise.contribsys.com/gems/sidekiq-ent-2.5.3.gem
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0^M  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    >
curl: (22) The requested URL returned error: 401
error: cannot download sidekiq-ent-2.5.3.gem from any mirror

This is saying to me that the bundle config isnt being used, but I've tried it using local config, global config, and envvars and its the same thing (and like I said, was working previously).

Any ideas on how to debug it?

EDIT: Ive also confirmed I can start a dev shell with required native gem dependencies and run a native bundle install successfully.

[inscapist]

This is probably unrelated to ruby-nix. But does it work now if you try to regenerate gemset.nix from scratch?

echo '{}' >! gemset.nix
nix run github:inscapist/bundix

[J-Swift]

Same issue, but if I modify gemset.nix to hardcode the credentials in the url it works. Its also weird that the gemset.nix is generated with a sha256 with the 'bad' source.remotes.. I assume that means it fetched it successfully when generating?

ie from:

source.remotes = ["https://enterprise.contribsys.com"];

to

source.remotes = ["https://username:password@enterprise.contribsys.com"];

[inscapist]

Same issue, but if I modify gemset.nix to hardcode the credentials in the url it works. This would imply that the issue is not from ruby-nix. I have checked the upstream buildRubyGem (that does the actual fetching) in nixpkgs and there were no changes recently.

Since you said you tried bundle config and envvars to no avail, I think we can suspect that something might've changed on the server side.

Also, I assume you did not make any changes to flake.lock (nix flake update) before the error happens?

Its also weird that the gemset.nix is generated with a sha256 with the 'bad' source.remotes..

sorry, what do you mean by that?

[J-Swift]

I reached out to a both mike at sidekiq directly, and also general audience on a rails slack and I seem to be the only one so I suspect its nix-specific. That plus the ability to curl the endpoint, and install via normal bundler as well.

sorry, what do you mean by that?

I havent looked into the gemset.nix generation, but I assume the sha256 is the result of actually fetching the gem right? If so, then its weird if bundix can run, and then running nix develop for a devshell using that gemset.nix is what is failing.

[J-Swift]

OK well this just randomly went away this week 😓

My best guess is some combination of specific nix version and my local machine... I had another weird issue with installing VSCode from home-manager that also resolved itself.

[inscapist]

could be your network (my guess is VPN?)