x/pkgsite: API for pkg.go.dev

[rhcarvalho]

Prior to pkg.go.dev, godoc.org has had a JSON API that can be used to, among other things, discover importers of a given package.

Example: https://api.godoc.org/importers/golang.org/x/net/html

Given that pkg.go.dev does a much better job at tracking importers thanks to Go Modules and the Module Proxy, it would be nice if the community could get access to a public API similar to that of godoc.org.

[gwatts]

I use the Dash doc viewer on my Mac which i find super useful for building a local index of all sorts of docs, Go included - Just found i couldn't install a new Go docset into it, presumably as it can no longer build an index from api.godoc.org.

Would be nice to point the author towards an alternate index if this one isn't returning as the usefulness of Dash is greatly diminished for me without having rapid access to the docs of Go packages i use frequently.

[AlekSi]

Dash 6.3.1 now relies on the web scraping. It works, but API would be better.

[hyangah]

FYI deps.dev launched a new API service.

https://docs.deps.dev/api/v3alpha/

Its GetPackage, GetVersion, GetDependencies may be useful for users who need the list of versions, license info, and dependency info. I think it also provides vulnerability info as well. (cc @adg)

This does not cover the /search, /packages, /importers endpoints of the old gddo though. (OTOH, it's unclear to me what people want to see from /packages endpoint given the current volume of data)

[avamsi]

I was looking for an API to search for Go "commands" -- https://deps.dev/_/search/suggest?system=go&kind=package&q=axl (and then https://api.deps.dev/v3alpha/systems/go/packages/github.com%2Favamsi%2Faxl) kinda works (I don't mind that deps.dev/_/search is hacky and potentially unstable), but pkg.go.dev also annotates the commands as such (see https://pkg.go.dev/search?q=axl, for example), so it's easy to tell them apart from vanilla libraries.

[sfblackl-intel]

https://docs.deps.dev/api/v3alpha/

We are using that...but realized that for Go packages in particular, there is no publish date...which led me to this thread. My main real ask is that publish date be included for Go packages...otherwise my solution appears to be to call something like this for each package/version (https://proxy.golang.org/<module>/@v/v1.9.0.info).

[dvob]

I also used api.godoc.org for my project https://github.com/dvob/go-project-usage/ To obtain the importers I now scrape https://pkg.go.dev/%s?tab=importedby (see https://github.com/dvob/go-project-usage/blob/72c5625c361b7e81fa2d1a35591c30830dc1164f/main.go#L134). Unfortunately this is prone to break (e.g if CSS class name changes) and for packages with many importers it only returns 20k packages.

[xnuinside]

I want to woke up this tread. https://docs.deps.dev/ that is referencing before does not contain any information about standard libraries packages (https://pkg.go.dev/std). Answering on why and who will use API: all researchers & data miners that needed to analyze open source packages (huge set of tools for code analyzes, code efficiency & vulnerabilities) (and it is only from my work-domain, I believe that there are more purposes) - we need information about package when we resolve vulnerabilities, when we investigate how languages growth, that dynamic on them, we need this information to track new releases & fixes & etc. Mostly all popular ecosystems for languages (maven, pypi, crates, conan, conda, etc) & operation systems like linux provide API to get this information - it could be DB dump, it could be REST API, github repo, anything, it could be huge csv files - any format will be better when scraping website.

UPD: also, there is no ANY informations about packages (subpackages?) inside repo, for example, https://pkg.go.dev/vuln/GO-2021-0073 for Affected Package - github.com/git-lfs/git-lfs/lfsapi on deps.dev info only about 'root' packages like github.com/git-lfs/git-lfs and you cannot find any information that packages inside

[mark-pictor-csec]

Another use for the API would be to check if any versions of a module are retracted. Right now the only way I see to get that info is pretty terrible - I must GET pkgs.go.dev/path/to/pkg/?tab=versions, and search the html for the word retracted. If it's there then I get to parse the html and figure out which version(s) are so marked.

People at my company are very thankful that go vulnerabilities usually include affected symbols (example), as it allows us to report to customers whether they're actually using a vulnerable function with far better accuracy.

So while vuln reports are much better in go, for retractions we must parse a web page and hope the html doesn't change. This feels like a real step back.

I hope an API can be prioritized.