purlLookupBatch Fails Entire Batch If Any Repo Is Missing

[nathannaveen]

Hi, we at GUAC https://github.com/guacsec/guac are trying to use new API, deps.dev/api/v3alpha, but there seems to be an issue with the purlLookupBatch.

If any of the repos passed to the batch request aren't contained in the deps.dev database, the entire batch query returns nil.

So the only way to check whether a purl is contained in the deps.dev database is to manually do a purlLookup for each individual purl. If we do this, we don't really need to use the purlLookupBatch functionality.

[sarnesjo-google]

Hi @nathannaveen!

Do you have an example of a batch that reproduces this issue?

Also, please note that not all purl types are supported (docs):

Supported values for type are cargo, golang, maven, npm, nuget and pypi.

[nathannaveen]

Hey @sarnesjo-google, I apologize for the late reply.

So, an example of a batch that fails is using two purls (Both of which have a type of Golang, so they are supported types):

• The purl pkg:golang/github.com/gofrs/uuid@v4.0.0+incompatible is found in deps dev
• While the purl pkg:golang/[github.com/google/gnostic@v0.5.7-v3refs is not found in deps dev.

So, if we do the batch request with both purls, we get a returned value of nil:

batchReq := &pb.PurlLookupBatchRequest{
    Requests: []*pb.PurlLookupRequest{
        {
            Purl: "pkg:golang/github.com/gofrs/uuid@v4.0.0+incompatible",
        },
        {
                        // Comment out this to get a non-nil response
            Purl: "purl pkg:golang/github.com/google/gnostic@v0.5.7-v3refs",
        },
    },
}

batchResp, err := d.client.PurlLookupBatch(ctx, batchReq)
if err != nil {
    fmt.Printf("failed to lookup purls in batch: %v", err)
    return
}
fmt.Printf("batch response: %v", batchResp)

---

But, if we do a similar batch request but, only with the purl contained in deps dev, we get back our expected result:

responses:{request:{purl:\"pkg:golang/github.com/gofrs/uuid@v4.0.0+incompatible\"} result:{version:{version_key:{system:GO name:\"github.com/gofrs/uuid\" version:\"v4.0.0+incompatible\"} purl:\"pkg:golang/github.com/gofrs/uuid@v4.0.0%2Bincompatible\" licenses:\"MIT\" links:{label:\"SOURCE_REPO\" url:\"https://github.com/gofrs/uuid\"} related_projects:{project_key:{id:\"github.com/gofrs/uuid\"} relation_provenance:GO_ORIGIN relation_type:SOURCE_REPO} 11:\"\\n\\x15github.com/gofrs/uuid\\x12\\x13v4.0.0+incompatible\\x1a\\x0eGO_MODULE_FILE\" 13:\"\\n\\x03MIT\\x12\\x03MIT\"}}}

[sarnesjo-google]

You have a typo in your second purl:

Purl: "purl pkg:golang/github.com/google/gnostic@v0.5.7-v3refs",
       ^^^^^

This is reported by the API:

$ grpcurl -d '{"requests":[{"purl":"pkg:golang/github.com/gofrs/uuid@v4.0.0+incompatible"},{"purl":"purl pkg:golang/github.com/google/gnostic@v0.5.7-v3refs"}]}' api.deps.dev:443 deps_dev.v3alpha.Insights/PurlLookupBatch
ERROR:
  Code: InvalidArgument
  Message: invalid purl "purl pkg:golang/github.com/google/gnostic@v0.5.7-v3refs" at request index 1