fetching package urls from api instead of hardcoded text file - Githubissues

RomanVolkov / go.get.nvim

2 stars 0 forks source link

fetching package urls from api instead of hardcoded text file #1

Open mAmineChniti opened 1 week ago

mAmineChniti commented 1 week ago

to avoid limitations that come with files, changing packages and ones that are newly published daily there's an api that you can just query to get the url and all the information you need about a go package

https://github.com/golang/go/issues/36785#issuecomment-1504550554

https://docs.deps.dev/api/v3alpha/

RomanVolkov commented 1 week ago

@mAmineChniti hey! thanks for sharing, that is actually interesting. I will take a look if I can fully replace with this API

RomanVolkov commented 1 week ago

@mAmineChniti at least at the main page this is exactly what I wanted to find when I researched for this project. so on the main page: https://deps.dev - there is actual search by substring that returns list of matches. but they don't use the API directly, the have different endpoint https://deps.dev/_/search/suggest?q=gon&system=GO&kind=PACKAGE

and I checked api docs and still cannot understand which I could use to do the search. Maybe you have an idea? /GetPackage needs full name /Query does not return anything without versionKey.version, I tried to send only name and system - no results even for examples.

Of course I can use first endpoint from deps.dev, but I don't think it's a good idea to use undocumented api :)

RomanVolkov commented 1 week ago

okay, I messed with this API more and not sure what actually I can get from it.

I was not able to implement search by substring (but maybe I miss something)
I am not sure how I can use e.g. GetPackage to understand if a packages was renamed. E.g., chi router used to be github.com/pressly/chi. If you try to go get it - it will fail
```
module declares its path as: github.com/go-chi/chi
but was required as: github.com/pressly/chi
```

It would be ideal if I can get implement search with it, that will make a plugin much easier to implement (and expand for python as well), but so far I cannot understand if I can do that with this api

mAmineChniti commented 5 days ago

Maybe you should make github itself your index, search by repo names eliminate forks and non go language repos, that's the only solution i can think about, it's really shit that there's no go packages registry api, npm has one, rust's crates too

mAmineChniti commented 5 days ago

Also users should deal with the renamed packages errors by themselves or if possible you can parse the error and just go get the new one

RomanVolkov commented 3 days ago

@mAmineChniti I actually returned to original approach, but I used deps.dev too So now I have 2 working checks after I parse proxy.go updates

Remove all forks. Most of packages (like 90-95%) are hosted on github and it was easy to understand if publised package is just a fork. If it is - I ignore them and do not add to index
Thanks to you @mAmineChniti I'm using deps.dev API to filter all misspelled updated. A lot of cases when packages was published with some type in the name (uppercased vs lowercased). And I managed to filter that with GetPackage endpoint
The last one is the most good, but I had to disabled it: https://github.com/RomanVolkov/go.get.nvim/blob/main/index_builder/validations/package.go Ideal that I actually trying to download and install a package into an empty project. If it fails - discard the url. And it works really good. But the problem is that even with batching I hit "not enough space on runner" error. I managed to run it without errors with batch == 200 concurrent installations, but it's very slow. I won't fit into 6h limit for 1 week update of proxy.go. Si I disabled then since I cannot use much bigger runners here as they cost additionally, maybe if this plugin will get some support - then probably yes. Another option would be to run it daily, but I think it's too much of commits into the repo just with index updates.

So now index is being update once in a week collecting all the updates first and then filtering them with (1) and (2).

RomanVolkov commented 3 days ago

@mAmineChniti problem with renamed packages that you cannot understand if a packages was moved to another repo. There is no connection. The only way to filter this is option (3)

mAmineChniti commented 3 days ago

yeah 3) is the whole problem with this approach, like having a hardcoded text file and having to parse and check for packages on your own is very unmaintainable is the long run, but there's no other way around it until the official go people create an api for their registry, ayy good luck on this project i'll be using it once it can be installed on lazy.nvim

RomanVolkov commented 3 days ago

@mAmineChniti but you can actually install it already :) at least I tested with lazy and it worked locally. But I had a local version too, so I would appreciate if you test it and tell me if it worked

RomanVolkov commented 3 days ago

@mAmineChniti in https://github.com/RomanVolkov/go.get.nvim readme there is an information how to call the plugin.

mAmineChniti commented 3 days ago

would be nice if you can include like a lazy.nvim config in the readme

RomanVolkov commented 3 days ago

@mAmineChniti will do👍 I actually want to test other package managers, but I used mostly lazy.nvim with neovim.

RomanVolkov commented 3 days ago

@mAmineChniti https://github.com/RomanVolkov/go.get.nvim added