clearlydefined / crawler

A service that crawls projects and packages for information relevant to ClearlyDefined
MIT License
43 stars 30 forks source link

Error encountered when fetch Go packages without version specified #565

Open yashkohli88 opened 1 month ago

yashkohli88 commented 1 month ago

Description

The _getLatestVersion function in the goFetch.js file is designed to fetch the latest version of a Go package when no specific version is provided. However, we are encountering an error when attempting to use this function with the crawler for Go packages. It seems that the URL being used to retrieve the latest package version is invalid, causing the process to fail.

Expected Behavior

When a Go package is specified without a version, the crawler should automatically retrieve information for the latest version of the package using a correct URL.

Current Behavior

The crawler fails with the below error when trying to fetch the latest version of a Go package due to an invalid URL.

RequestError: Error: getaddrinfo ENOTFOUND golang

Steps to Reproduce

  1. Invoke the crawler for a Go package without specifying a version. We are using go/golang/rsc.io/quote coordinate .
  2. Observe that the crawler fails to process the request and returns the below error RequestError: Error: getaddrinfo ENOTFOUND golang

Solution

The URL used in the _getLatestVersion function needs to be corrected as below.

Current URL used - const initial_url = 'https://${spec.provider}/${spec.namespace}/${spec.name}/@v/list' where the value of spec.provider is coming to be golang which forms an incorrect URL.

Updated Working URL should be - const initial_url = 'https://proxy.golang.org/${spec.namespace}/${spec.name}/@v/list'