Why does v0.6.0 query a custom index domain-name, instead of crates.io API ?

[BarbossHack]

Is your feature request related to a problem? Please describe.

It is not possible to use "Crates" offline anymore (since v0.6.0), as it have to query https://api.crates-vsc.space

Describe the solution you'd like

Instead of getting a crate version (for example base64) from https://api.crates-vsc.space/index/versions/base64 , it should query https://crates.io/api/v1/crates/base64/versions , you will have the same results (see the screenshot just bellow), and we will able to deploy our own offline crates.io API if needed

Or at least, have the same api URI than crates.io (/crates/NAME/versions and not /index/versions/NAME), and same json response, to allow us to set our own offline api domain name

Thank you :+1:

[kristof-mattei]

I think this is actually quite concerning.

All of the sudden we have data being send to a 3rd party URL. There are no release notes, except for all cis migration done, we don't know the owner and we don't know what happens with our data.

The WHOIS shows a registrant in Iceland (IS):

https://www.whois.com/whois/crates-vsc.space

...
Registrant Organization: Privacy service provided by Withheld for Privacy ehf
Registrant State/Province: Capital Region
Registrant Country: IS
Registrant Email: Please query the RDDS service of the Registrar of Record identified in this output for information on how to contact the Registrant, Admin, or Tech contact of the queried domain name.
...

[BarbossHack]

Release notes are here https://github.com/serayuzgur/crates/blob/master/CHANGELOG.md, but yeah it wasn't done the usual way (not released on github), and I still see no reason not to use the official API

And because of this, people will think it fetches crates.io, but it obviously doesn't : https://github.com/serayuzgur/crates/blob/84b70c44df439ae83633a6197c67214296542afb/src/core/fetcher.ts#L12

[serayuzgur]

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead. So, we did that, but then people started complaining about API limits.

To solve that problem, we added local index support, thinking it'd make things better. But then folks started saying cargo wasn't getting updated, and we couldn't see the latest versions or have a local registry.

Eventually, cargo went through a big protocol change, which was a major bummer for us. So, I took matters into my own hands and created a similar API using the crates.io GitHub index.

Why did I do it? Well, managing that local index was driving me nuts, and I simply didn't have enough time to deal with all the user issues. Plus, that protocol change was a real dealbreaker. And don't even get me started on how huge the extension code was getting!

The only info we send to the server is the crate name you're requesting. Don't worry; it's all open source, and you can check out the data we send—it's transparent like that. I got the cheapest domain because, honestly, nobody's paying or donating, and the privacy option that came with it was just the default, so I didn't bother changing it or thinking too much about it.

[BarbossHack]

Thank you for your explanations @serayuzgur :+1:

Quote from my first comment :

Or at least, have the same api URI than crates.io (/crates/NAME/versions and not /index/versions/NAME), and same json response, to allow us to set our own offline api domain name

Would that be possible ?

Thank for your work

[BarbossHack]

or should I open a new issue ?

[BarbossHack]

(Also if you publish your server api on github I would be able to make a PR to help you)

[kristof-mattei]

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead. So, we did that, but then people started complaining about API limits.

To solve that problem, we added local index support, thinking it'd make things better. But then folks started saying cargo wasn't getting updated, and we couldn't see the latest versions or have a local registry.

Eventually, cargo went through a big protocol change, which was a major bummer for us. So, I took matters into my own hands and created a similar API using the crates.io GitHub index.

Why did I do it? Well, managing that local index was driving me nuts, and I simply didn't have enough time to deal with all the user issues. Plus, that protocol change was a real dealbreaker. And don't even get me started on how huge the extension code was getting!

The only info we send to the server is the crate name you're requesting. Don't worry; it's all open source, and you can check out the data we send—it's transparent like that. I got the cheapest domain because, honestly, nobody's paying or donating, and the privacy option that came with it was just the default, so I didn't bother changing it or thinking too much about it.

I appreciate the update.

With all of that said and crates.io now being sparse, is this still an issue?

[serayuzgur]

or should I open a new issue ?

No, there's no need for that. I don't plan to make them identical. You see, the crates.io API isn't quite designed for the specific task of version listing; it generates a response JSON that's quite bloated with unnecessary data. Given this situation, I've decided to pursue an alternative course of action.

[BarbossHack]

Advantage of Json is that you don't have to add all unnecessary fields to make it compatible, just adding the versions under the same tree is sufficient.

Thank you for your answer.

[BarbossHack]

Or at least could you share your api server code on github ? So we will be able to deploy our own server (for offline uses for example).

Thank you.

[serayuzgur]

Or at least could you share your api server code on github ? So we will be able to deploy our own server (for offline uses for example).

Thank you.

What is the rationale behind requiring offline usage if you're unable to access the latest version information and install the desired version of the crate you seek?

I am planning some scenarios for the server , In progress.

[BarbossHack]

What is the rationale behind requiring offline usage if you're unable to access the latest version information and install the desired version of the crate you seek?

I have a full crates.io mirror offline which is often updated, so I'm able to install latest crates version. If you publish your server code on github, I will be able to deploy my own server offline too, using this mirror.

[serayuzgur]

I hope it is a joke , a bad one I guess.

On Thu, Aug 3, 2023 at 3:24 PM pingufreak @.***> wrote:

Why is the domain hosted in Island while you are living in Turkey? For me it looks also a bit strange. Identity theft?

— Reply to this email directly, view it on GitHub https://github.com/serayuzgur/crates/issues/197#issuecomment-1663889175, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCZTMQMFMWC2FTUTTKA5DDXTOJ7HANCNFSM6AAAAAA2LFQNTM . You are receiving this because you were mentioned.Message ID: @.***>

[pingufreak]

Okay, here's the whole deal: At first, we were using the crates.io API, but they asked us not to and switch to the GitHub API instead.

May I ask you who declined to use crates.io? Namecheap is blocked in a lot of security appliances. For instance at PaloAlto the url https://api.crates-vsc.space/ is categorized as insufficient-content. If it would be possible to get it back to crates.io, it would be very nice.

Thank you

[serayuzgur]

@pingufreak

May I ask you who declined to use crates.io?

I can't recall the individual responsible for managing the crates.io API. 3-4 years ago.

Namecheap is blocked in a lot of security appliances.

I can transfer dns to Digital Ocean. Will it function properly?

the url https://api.crates-vsc.space/ is categorized as insufficient-content

Maybe you can help me to make url trustable. I don't understand why it is considered like that.

[pingufreak]

Why is this closed? If you can change it back to crates.io it would be the best. :)

[BarbossHack]

Why is this closed? If you can change it back to crates.io it would be the best. :)

It was closed because he didn't wanted to uses crates.io, mainly for optimisation reasons (see https://github.com/serayuzgur/crates/issues/195#issuecomment-1640801531) (imho the only valid reason)

[serayuzgur]

Why is this closed? If you can change it back to crates.io it would be the best. :)

It was closed because he didn't wanted to uses crates.io, mainly for optimisation reasons (see #195 (comment)) (imho the only valid reason)

In addition several other minor issues that do not affect your life but do affect mine 😄 . Base for bigger features like private registry support, on premise crates index server, etc.

[WUST-mengqinyu]

But I have problem with this host. Is there any alternative？ Completed with errors xxx: Error: statusCode=504

[junderw]

Just thought I'd link to a discussion regarding this issue.

https://www.reddit.com/r/rust/comments/15ye9he/i_forked_crates_vscode_extension_to_make_it_use/

Seems as though a fork of this project is using the sparse HTTP protocol to get the info directly from crates.io's AWS S3 files directly (which is much easier on them because it doesn't even touch their API)

Would be nice if that could be merged. Not that I don't trust the new author, but it seems like post-sparse-protocol this is the optimal method for doing this.

[BarbossHack]

Would be nice if that could be merged. Not that I don't trust the new author, but it seems like post-sparse-protocol this is the optimal method for doing this.

(Don't worry you're totally right, and if this get merged I would even delete my fork :+1: )

[junderw]

https://github.com/serayuzgur/crates/compare/84b70c44df439ae83633a6197c67214296542afb...BarbossHack:crates-io:2ab0c2f649b9037ca2e3717a0592eac1dfecb94b

The diff seems pretty small. Perhaps preparing a PR for this repo (without all the funding removals and renamings etc.) would be appropriate.

[BarbossHack]

see https://github.com/serayuzgur/crates/pull/204

I know that @serayuzgur has other reasons not to use crates.io (whether with the api or sparse index, see https://github.com/serayuzgur/crates/issues/195#issuecomment-1640801531 and https://github.com/serayuzgur/crates/issues/197#issuecomment-1671600461), so I'm quite sure this PR will be closed, but at least I tried. And again, I understand these reasons (and I'm thankful for your work @serayuzgur), but I needed an offline compatible version...

[simonsan]

see #204

I know that @serayuzgur has other reasons not to use crates.io (whether with the api or sparse index, see #195 (comment) and #197 (comment)), so I'm quite sure this PR will be closed, but at least I tried. And again, I understand these reasons (and I'm thankful for your work @serayuzgur), but I needed an offline compatible version...

Thanks for your work! It would be nice if serayuzgur would merge this and maybe there is a world, where both solutions can coexist, in the sense of making the crates index configurable in the settings?

[serayuzgur]

@simonsan I will think about it. I need to calculate the effect of this on new planned features

[serayuzgur]

Since the PR is replacing the code but not implementing a different api connection code regarding to configuration I can not accept it. @BarbossHack Thanks for your effort, please take a look at my comments on PR

[serayuzgur]

@BarbossHack I'd kindly request that you consider removing it from the market altogether. Given that the fork no longer offers offline support due to the shift towards a sparse approach, it might inadvertently mislead users who are seeking that functionality. This step would ensure that users make informed decisions and that we avoid any potential confusion.

[simonsan]

As a long-time user of crates-io extension, I would appreciate, if the fork would stay up at least until this issue is resolved and the PR (with the additional feature of choosing the index url in the settings) is merged. When I read about this issue here, I immediately switched to the fork, as I'm in general pretty concerned about my privacy.

[serayuzgur]

As a long-time user of crates-io extension, I would appreciate, if the fork would stay up at least until this issue is resolved and the PR (with the additional feature of choosing the index url in the settings) is merged. When I read about this issue here, I immediately switched to the fork, as I'm in general pretty concerned about my privacy.

@simonsan If so , please use older version of crates or use packaged version of the fork. I am not feeling good about it all. Regarding privacy I'm making an effort to provide reasonable justifications, but maintaining my stance is becoming more difficult. Please explain how the crate name being the only information transmitted to the server can cause you to be concerned about privacy.

[BarbossHack]

@BarbossHack I'd kindly request that you consider removing it from the market altogether. Given that the fork no longer offers offline support due to the shift towards a sparse approach, it might inadvertently mislead users who are seeking that functionality. This step would ensure that users make informed decisions and that we avoid any potential confusion.

I am currently using my fork offline, so it still allow offline compatibility for users. I have a sparse index clone offline (git clone) with fake dns, so crates-io is able to fetch it (as it would with crates.io api, I would have used the same process). Maybe it wasn't clear, but when I mean "offline" I mean "on a network that does not have internet connection", not simply "local fetch"

I saw your comments on PR, I will be able to make the changes soon (can't do it at work), thank you for allowing us to implement this feature 👍

[serayuzgur]

published v 0.6.2

[serayuzgur]

@BarbossHack please check changelog

[BarbossHack]

@BarbossHack please check changelog

Thank you for the credits 🙏 🚀