search

FiloSottile / gvt

gvt was a minimal go vendoring tool, based on gb-vendor. Today, you want to use modules instead.
MIT License
729 stars 67 forks source link

recent gvt pulls in unused dependencies #56

Closed jsravn closed 8 years ago

jsravn commented 8 years ago

gvt has been working great for us, but a recent update causes it to pull in far more dependencies recursively than it should.

For example, older gvt:

$ gvt fetch github.com/aws/aws-sdk-go/service/elb
2016/05/31 14:50:24 fetching recursive dependency github.com/aws/aws-sdk-go/aws
2016/05/31 14:50:25 fetching recursive dependency github.com/aws/aws-sdk-go/private/protocol/query
2016/05/31 14:50:26 fetching recursive dependency github.com/aws/aws-sdk-go/private/protocol
2016/05/31 14:50:27 fetching recursive dependency github.com/aws/aws-sdk-go/private/signer/v4
2016/05/31 14:50:28 fetching recursive dependency github.com/aws/aws-sdk-go/private/waiter
2016/05/31 14:50:28 fetching recursive dependency github.com/go-ini/ini
2016/05/31 14:50:30 fetching recursive dependency github.com/jmespath/go-jmespath

Most recent gvt:

$ gvt fetch github.com/aws/aws-sdk-go/service/elb
2016/05/31 14:52:34 Fetching: github.com/aws/aws-sdk-go/service/elb
2016/05/31 14:52:39 · Fetching recursive dependency: github.com/aws/aws-sdk-go/aws/client
2016/05/31 14:52:39 ·· Fetching recursive dependency: github.com/aws/aws-sdk-go/aws
2016/05/31 14:52:39 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/awstesting/unit
2016/05/31 14:52:39 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/vendor/github.com/go-ini/ini
2016/05/31 14:52:39 ···· Fetching recursive dependency: github.com/smartystreets/goconvey/convey
2016/05/31 14:52:45 ····· Fetching recursive dependency: github.com/smartystreets/assertions
2016/05/31 14:52:47 ······ Fetching recursive dependency: golang.org/x/net/context
2016/05/31 14:52:48 ····· Fetching recursive dependency: github.com/jtolds/gls
2016/05/31 14:52:49 ······ Fetching recursive dependency: github.com/gopherjs/gopherjs/js
2016/05/31 14:52:51 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/sts
2016/05/31 14:52:51 ···· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ···· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/signer/v4
2016/05/31 14:52:51 ····· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol/rest
2016/05/31 14:52:51 ····· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/s3
2016/05/31 14:52:51 ······ Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ······ Fetching recursive dependency: github.com/aws/aws-sdk-go/private/waiter
2016/05/31 14:52:51 ······· Fetching recursive dependency: github.com/aws/aws-sdk-go/awstesting
2016/05/31 14:52:51 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/iam
2016/05/31 14:52:51 ········· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol/query
2016/05/31 14:52:51 ·········· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/util
2016/05/31 14:52:51 ··········· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
2016/05/31 14:52:51 ············ Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol
2016/05/31 14:52:51 ············· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudfront
2016/05/31 14:52:51 ·············· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ············· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/dynamodb
2016/05/31 14:52:51 ·············· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ············· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/elastictranscoder
2016/05/31 14:52:51 ············· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/ec2
2016/05/31 14:52:51 ·············· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ·············· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/endpoints
2016/05/31 14:52:51 ··············· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ············· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ·········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/sqs
2016/05/31 14:52:51 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ········ Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:51 ········ Fetching recursive dependency: github.com/lsegal/gucumber
2016/05/31 14:52:53 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:53 ········· Fetching recursive dependency: github.com/shiena/ansicolor
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/apigateway
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/glacier
2016/05/31 14:52:54 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudtrail
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cognitosync
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/configservice
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/ssm
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/elasticache
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/kinesis
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/opsworks
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudhsm
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/codedeploy
2016/05/31 14:52:54 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/directoryservice
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/emr
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/iot
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/workspaces
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudwatchevents
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/inspector
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/acm
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/directconnect
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/lambda
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/iotdataplane
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/elasticsearchservice
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/swf
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/route53
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/support
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/datapipeline
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/waf
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudsearchdomain
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/applicationdiscoveryservice
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cognitoidentity
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/firehose
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/marketplacecommerceanalytics
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/machinelearning
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/mobileanalytics
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/ecs
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/kms
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudsearch
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudwatchlogs
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/codecommit
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/simpledb
2016/05/31 14:52:55 ········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/signer/v2
2016/05/31 14:52:55 ·········· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/elasticbeanstalk
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/ses
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/storagegateway
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/autoscaling
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudformation
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/rds
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/redshift
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/cloudwatch
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/codepipeline
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/devicefarm
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/dynamodbstreams
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/efs
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/ecr
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/sns
2016/05/31 14:52:55 ········ Fetching recursive dependency: github.com/aws/aws-sdk-go/service/route53domains
2016/05/31 14:52:55 ······· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:55 ······ Fetching recursive dependency: github.com/stretchr/testify/require
2016/05/31 14:52:57 ······· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:57 ····· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:57 ··· Skipping (existing): github.com/stretchr/testify/assert
2016/05/31 14:52:57 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/vendor/github.com/jmespath/go-jmespath
2016/05/31 14:52:57 ···· Skipping (existing): github.com/stretchr/testify/assert

It pulls in the entire aws sdk now.

gsf commented 8 years ago

Just noticed this on a fetch of go-geom, where sqlmock is only used in tests:

$ gvt fetch github.com/twpayne/go-geom
2016/05/31 14:10:05 Fetching: github.com/twpayne/go-geom
2016/05/31 14:10:05 · Fetching recursive dependency: gopkg.in/DATA-DOG/go-sqlmock.v1
2016/05/31 14:10:06 ·· Fetching recursive dependency: github.com/DATA-DOG/go-sqlmock
2016/05/31 14:10:07 ··· Fetching recursive dependency: github.com/kisielk/sqlstruct
2016/05/31 14:10:07 · Fetching recursive dependency: github.com/twpayne/go-kml
ericchiang commented 8 years ago

It looks like gvt is fetching the dependencies of all sub packages of the repo.

For instance

$ gvt fetch golang.org/x/oauth2
2016/06/16 10:13:41 Fetching: golang.org/x/oauth2
2016/06/16 10:13:42 · Fetching recursive dependency: google.golang.org/cloud/compute/metadata
2016/06/16 10:13:45 ·· Fetching recursive dependency: golang.org/x/net/context
( continues to download the rest of the internet )

In this case gvt is pulling in dependencies of golang.org/x/oauth2/google

Would it be reasonable for gvt to ignore sub packages?

FiloSottile commented 8 years ago

Looks like this is #28 (open) + #63 (closed), am I right?

FiloSottile commented 8 years ago

Yep, fixing #63 fixed this to the extent that is possible until we start not fetching subpackages:

2016/07/28 03:11:45 Fetching: github.com/aws/aws-sdk-go/service/elb
2016/07/28 03:11:57 · Fetching recursive dependency: github.com/aws/aws-sdk-go/aws/client
2016/07/28 03:11:57 ·· Fetching recursive dependency: github.com/aws/aws-sdk-go/aws/request
2016/07/28 03:11:57 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/aws/awserr
2016/07/28 03:11:57 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/aws/awsutil
2016/07/28 03:11:57 ···· Fetching recursive dependency: github.com/aws/aws-sdk-go/vendor/github.com/jmespath/go-jmespath
2016/07/28 03:11:57 ··· Fetching recursive dependency: github.com/aws/aws-sdk-go/aws
2016/07/28 03:11:57 ···· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/endpoints
2016/07/28 03:11:57 ···· Fetching recursive dependency: github.com/aws/aws-sdk-go/vendor/github.com/go-ini/ini
2016/07/28 03:11:57 ···· Fetching recursive dependency: github.com/aws/aws-sdk-go/service/sts
2016/07/28 03:11:57 ····· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol/query
2016/07/28 03:11:57 ······ Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
2016/07/28 03:11:57 ······· Fetching recursive dependency: github.com/aws/aws-sdk-go/private/protocol
2016/07/28 03:11:57 · Fetching recursive dependency: github.com/aws/aws-sdk-go/private/waiter