unknown error: object not found - Githubissues

src-d / gitbase

SQL interface to git repositories, written in Go. https://docs.sourced.tech/gitbase

Apache License 2.0

2.06k stars 124 forks source link

unknown error: object not found #797

Closed eiso closed 5 years ago

eiso commented 5 years ago

When I run the following query I receive an error. This only happens once I join the blobs table. I am running this over 155 repos, if you need me to narrow it down, let me know.

SELECT file_path, blob_content
FROM commit_files
NATURAL JOIN refs
NATURAL JOIN blobs
WHERE ref_name='HEAD' AND LANGUAGE(file_path) = 'Go'
LIMIT 10

error:

ERROR 1105 (HY000): unknown error: object not found

mcuadros commented 5 years ago

This looks like more like a go-git bug, that will be hard to debug

erizocosmico commented 5 years ago

Could you send me the list of repos you're testing this against? Are they siva files or regular repositories? Which version of gitbase are you using? Is there any possibility your dataset contains a shallow repository?

I've tried on a few sets of repositories and cannot reproduce.

erizocosmico commented 5 years ago

Also, your query will be faster rewritten as:

SELECT file_path, blob_content
FROM refs
NATURAL JOIN commit_files
NATURAL JOIN files
WHERE ref_name='HEAD' AND LANGUAGE(file_path, blob_content) = 'Go'
LIMIT 10

commit_files is not squashed with blobs

This takes 0.04s on a warm cache, 0.07 without it on my dataset, the other query takes 13s on a warm cache, 1 min without it.

erizocosmico commented 5 years ago

This is the dataset

cloud.google.com/go/bigquery
cloud.google.com/go/bigtable
cloud.google.com/go/civil
cloud.google.com/go/cloudtasks
cloud.google.com/go/cmd
cloud.google.com/go/compute
cloud.google.com/go/container
cloud.google.com/go/dataproc
cloud.google.com/go/datastore
cloud.google.com/go/debugger
cloud.google.com/go/dialogflow
cloud.google.com/go/dlp
cloud.google.com/go/errorreporting
cloud.google.com/go/firestore
cloud.google.com/go/httpreplay
cloud.google.com/go/iam
cloud.google.com/go/internal
cloud.google.com/go/kms
cloud.google.com/go/language
cloud.google.com/go/logging
cloud.google.com/go/longrunning
cloud.google.com/go/monitoring
cloud.google.com/go/oslogin
cloud.google.com/go/profiler
cloud.google.com/go/pubsub
cloud.google.com/go/redis
cloud.google.com/go/rpcreplay
cloud.google.com/go/spanner
cloud.google.com/go/speech
cloud.google.com/go/storage
cloud.google.com/go/texttospeech
cloud.google.com/go/trace
cloud.google.com/go/translate
cloud.google.com/go/videointelligence
cloud.google.com/go/vision
github.com/acroca/go-symbols
github.com/ajstarks/svgo
github.com/alecthomas/template
github.com/alecthomas/units
github.com/apache/thrift
github.com/aws/aws-sdk-go
github.com/badgerodon/peg
github.com/bblfsh/tools
github.com/boltdb/bolt
github.com/bradfitz/gomemcache
github.com/BurntSushi/toml
github.com/cayleygraph/cayley
github.com/chrislusf/gleam
github.com/chrislusf/glow
github.com/cockroachdb/cockroach
github.com/colinmarc/hdfs
github.com/coreos/etcd
github.com/cweill/gotests
github.com/cznic/golex
github.com/cznic/mathutil
github.com/cznic/parser
github.com/cznic/sortutil
github.com/cznic/strutil
github.com/cznic/y
github.com/davecgh/go-spew
github.com/davidrjenni/reftools
github.com/dennwc/graphql
github.com/dennwc/okrs
github.com/derekparker/delve
github.com/dlclark/regexp2
github.com/dop251/goja
github.com/dustin/go-humanize
github.com/eapache/go-resiliency
github.com/eapache/go-xerial-snappy
github.com/eapache/queue
github.com/eiso/gleam
github.com/eiso/go-billy-siva
github.com/eiso/go-engine
github.com/eiso/go-engine-old
github.com/eiso/gpass
github.com/eiso/treeprint
github.com/emirpasic/gods
github.com/erizocosmico/gin-cache
github.com/fatih/color
github.com/fatih/gomodifytags
github.com/fsnotify/fsnotify
github.com/garyburd/redigo
github.com/gin-contrib/cors
github.com/gin-contrib/sse
github.com/gin-gonic/gin
github.com/glycerine/truepack
github.com/gocql/gocql
github.com/gogo/protobuf
github.com/go-kivik/couchdb
github.com/go-kivik/kivik
github.com/golang/glog
github.com/golang/lint
github.com/golang/protobuf
github.com/golang/snappy
github.com/google/go-github
github.com/google/go-querystring
github.com/googleapis/gax-go
github.com/gorilla/mux
github.com/go-sourcemap/sourcemap
github.com/go-sql-driver/mysql
github.com/go-stack/stack
github.com/grpc-ecosystem/grpc-opentracing
github.com/hailocab/go-hostpool
github.com/hashicorp/golang-lru
github.com/hashicorp/hcl
github.com/haya14busa/goplay
github.com/imdario/mergo
github.com/jackc/pgx
github.com/jbenet/go-context
github.com/jessevdk/go-flags
github.com/josharian/impl
github.com/juju/errors
github.com/julienschmidt/httprouter
github.com/kardianos/osext
github.com/kelseyhightower/envconfig
github.com/kevinburke/ssh_config
github.com/lann/builder
github.com/lann/ps
github.com/lib/pq
github.com/linkeddata/gojsonld
github.com/magiconair/properties
github.com/mailru/easyjson
github.com/Masterminds/glide
github.com/Masterminds/squirrel
github.com/mattes/migrate
github.com/mattn/go-colorable
github.com/mattn/go-isatty
github.com/mattn/go-runewidth
github.com/mcuadros/go-lookup
github.com/MichaelTJones/walk
github.com/mitchellh/go-homedir
github.com/mitchellh/hashstructure
github.com/mitchellh/mapstructure
github.com/nsf/gocode
github.com/nullbio/inflect
github.com/oklog/ulid
github.com/oliveagle/jsonpath
github.com/OneOfOne/xxhash
github.com/opentracing/opentracing-go
github.com/pborman/uuid
github.com/pelletier/go-buffruneio
github.com/pelletier/go-toml
github.com/peterh/liner
github.com/philhofer/fwd
github.com/pierrec/lz4
github.com/pingcap/tipb
github.com/pkg/errors
github.com/psilva261/timsort
github.com/ramya-rao-a/go-outline
github.com/rcrowley/go-metrics
github.com/robfig/go-cache
github.com/rogpeppe/godef
github.com/russross/blackfriday
github.com/satori/go.uuid
github.com/scritchley/orc
github.com/serenize/snaker
github.com/sergi/go-diff
github.com/Shopify/sarama
github.com/shurcooL/sanitized_anchor_name
github.com/sirupsen/logrus
github.com/skratchdot/open-golang
github.com/smacker/gleam
github.com/smacker/gleam-maxim
github.com/soheilhy/cmux
github.com/spf13/afero
github.com/spf13/cast
github.com/spf13/cobra
github.com/spf13/jwalterweatherman
github.com/spf13/pflag
github.com/spf13/viper
github.com/sqs/goreturns
github.com/src-d/core-retrieval
github.com/src-d/datasets
github.com/src-d/engine-cli
github.com/src-d/envconfig
github.com/src-d/gcfg
github.com/src-d/gitbase
github.com/src-d/gitbase-playground
github.com/src-d/go-git
github.com/src-d/go-kallax
github.com/src-d/go-siva
github.com/src-d/landing
github.com/streadway/amqp
github.com/stretchr/testify
github.com/syndtr/goleveldb
github.com/toqueteos/trie
github.com/tucnak/store
github.com/tylertreat/BoomFilters
github.com/uber/go-torch
github.com/ugorji/go
github.com/uudashr/gopkgs
github.com/vattle/sqlboiler
github.com/xanzy/ssh-agent
github.com/xitongsys/parquet-go
github.com/yanatan16/itertools
golang.org/x/crypto
golang.org/x/lint
golang.org/x/net
golang.org/x/oauth2
golang.org/x/sys
golang.org/x/text
golang.org/x/tools
google.golang.org/api/abusiveexperiencereport
google.golang.org/api/acceleratedmobilepageurl
google.golang.org/api/adexchangebuyer
google.golang.org/api/adexchangebuyer2
google.golang.org/api/adexchangeseller
google.golang.org/api/adexperiencereport
google.golang.org/api/admin
google.golang.org/api/adsense
google.golang.org/api/adsensehost
google.golang.org/api/analytics
google.golang.org/api/analyticsreporting
google.golang.org/api/androiddeviceprovisioning
google.golang.org/api/androidenterprise
google.golang.org/api/androidmanagement
google.golang.org/api/androidpublisher
google.golang.org/api/appengine
google.golang.org/api/appsactivity
google.golang.org/api/appstate
google.golang.org/api/bigquery
google.golang.org/api/bigquerydatatransfer
google.golang.org/api/blogger
google.golang.org/api/books
google.golang.org/api/calendar
google.golang.org/api/chat
google.golang.org/api/civicinfo
google.golang.org/api/classroom
google.golang.org/api/cloudbilling
google.golang.org/api/cloudbuild
google.golang.org/api/clouddebugger
google.golang.org/api/clouderrorreporting
google.golang.org/api/cloudfunctions
google.golang.org/api/cloudiot
google.golang.org/api/cloudkms
google.golang.org/api/cloudmonitoring
google.golang.org/api/cloudprofiler
google.golang.org/api/cloudresourcemanager
google.golang.org/api/cloudshell
google.golang.org/api/cloudtasks
google.golang.org/api/cloudtrace
google.golang.org/api/clouduseraccounts
google.golang.org/api/composer
google.golang.org/api/compute
google.golang.org/api/consumersurveys
google.golang.org/api/container
google.golang.org/api/content
google.golang.org/api/customsearch
google.golang.org/api/dataflow
google.golang.org/api/dataproc
google.golang.org/api/datastore
google.golang.org/api/deploymentmanager
google.golang.org/api/dfareporting
google.golang.org/api/dialogflow
google.golang.org/api/digitalassetlinks
google.golang.org/api/discovery
google.golang.org/api/dlp
google.golang.org/api/dns
google.golang.org/api/doubleclickbidmanager
google.golang.org/api/doubleclicksearch
google.golang.org/api/drive
google.golang.org/api/examples
google.golang.org/api/firebasedynamiclinks
google.golang.org/api/firebaseremoteconfig
google.golang.org/api/firebaserules
google.golang.org/api/firestore
google.golang.org/api/fitness
google.golang.org/api/fusiontables
google.golang.org/api/games
google.golang.org/api/gamesconfiguration
google.golang.org/api/gamesmanagement
google.golang.org/api/genomics
google.golang.org/api/gensupport
google.golang.org/api/gmail
google.golang.org/api/googleapi
google.golang.org/api/google-api-go-generator
google.golang.org/api/groupsmigration
google.golang.org/api/groupssettings
google.golang.org/api/iam
google.golang.org/api/iamcredentials
google.golang.org/api/iap
google.golang.org/api/identitytoolkit
google.golang.org/api/indexing
google.golang.org/api/integration-tests
google.golang.org/api/internal
google.golang.org/api/iterator
google.golang.org/api/jobs
google.golang.org/api/kgsearch
google.golang.org/api/language
google.golang.org/api/lib
google.golang.org/api/licensing
google.golang.org/api/logging
google.golang.org/api/manufacturers
google.golang.org/api/mirror
google.golang.org/api/ml
google.golang.org/api/monitoring
google.golang.org/api/oauth2
google.golang.org/api/option
google.golang.org/api/oslogin
google.golang.org/api/pagespeedonline
google.golang.org/api/partners
google.golang.org/api/people
google.golang.org/api/photoslibrary
google.golang.org/api/playcustomapp
google.golang.org/api/playmoviespartner
google.golang.org/api/plus
google.golang.org/api/plusdomains
google.golang.org/api/poly
google.golang.org/api/prediction
google.golang.org/api/proximitybeacon
google.golang.org/api/pubsub
google.golang.org/api/qpxexpress
google.golang.org/api/redis
google.golang.org/api/replicapool
google.golang.org/api/replicapoolupdater
google.golang.org/api/reseller
google.golang.org/api/resourceviews
google.golang.org/api/runtimeconfig
google.golang.org/api/safebrowsing
google.golang.org/api/script
google.golang.org/api/searchconsole
google.golang.org/api/servicebroker
google.golang.org/api/serviceconsumermanagement
google.golang.org/api/servicecontrol
google.golang.org/api/servicemanagement
google.golang.org/api/serviceusage
google.golang.org/api/serviceuser
google.golang.org/api/sheets
google.golang.org/api/siteverification
google.golang.org/api/slides
google.golang.org/api/sourcerepo
google.golang.org/api/spanner
google.golang.org/api/spectrum
google.golang.org/api/speech
google.golang.org/api/sqladmin
google.golang.org/api/storage
google.golang.org/api/storagetransfer
google.golang.org/api/streetviewpublish
google.golang.org/api/support
google.golang.org/api/surveys
google.golang.org/api/tagmanager
google.golang.org/api/taskqueue
google.golang.org/api/tasks
google.golang.org/api/testing
google.golang.org/api/texttospeech
google.golang.org/api/toolresults
google.golang.org/api/tpu
google.golang.org/api/tracing
google.golang.org/api/translate
google.golang.org/api/transport
google.golang.org/api/urlshortener
google.golang.org/api/vault
google.golang.org/api/videointelligence
google.golang.org/api/vision
google.golang.org/api/webfonts
google.golang.org/api/webmasters
google.golang.org/api/websecurityscanner
google.golang.org/api/youtube
google.golang.org/api/youtubeanalytics
google.golang.org/api/youtubereporting
google.golang.org/appengine/aetest
google.golang.org/appengine/blobstore
google.golang.org/appengine/capability
google.golang.org/appengine/channel
google.golang.org/appengine/cloudsql
google.golang.org/appengine/cmd
google.golang.org/appengine/datastore
google.golang.org/appengine/delay
google.golang.org/appengine/demos
google.golang.org/appengine/file
google.golang.org/appengine/image
google.golang.org/appengine/internal
google.golang.org/appengine/log
google.golang.org/appengine/mail
google.golang.org/appengine/memcache
google.golang.org/appengine/module
google.golang.org/appengine/remote_api
google.golang.org/appengine/runtime
google.golang.org/appengine/search
google.golang.org/appengine/socket
google.golang.org/appengine/taskqueue
google.golang.org/appengine/urlfetch
google.golang.org/appengine/user
google.golang.org/appengine/xmpp
google.golang.org/genproto/googleapis
google.golang.org/genproto/protobuf
google.golang.org/grpc/balancer
google.golang.org/grpc/benchmark
google.golang.org/grpc/channelz
google.golang.org/grpc/codes
google.golang.org/grpc/connectivity
google.golang.org/grpc/credentials
google.golang.org/grpc/Documentation
google.golang.org/grpc/encoding
google.golang.org/grpc/examples
google.golang.org/grpc/grpclog
google.golang.org/grpc/health
google.golang.org/grpc/internal
google.golang.org/grpc/interop
google.golang.org/grpc/keepalive
google.golang.org/grpc/metadata
google.golang.org/grpc/naming
google.golang.org/grpc/peer
google.golang.org/grpc/reflection
google.golang.org/grpc/resolver
google.golang.org/grpc/stats
google.golang.org/grpc/status
google.golang.org/grpc/stress
google.golang.org/grpc/tap
google.golang.org/grpc/test
google.golang.org/grpc/testdata
go.opencensus.io/examples/exporter
go.opencensus.io/examples/grpc
go.opencensus.io/examples/helloworld
go.opencensus.io/examples/http
go.opencensus.io/exporter/jaeger
go.opencensus.io/exporter/prometheus
go.opencensus.io/exporter/stackdriver
go.opencensus.io/exporter/zipkin
go.opencensus.io/internal/check
go.opencensus.io/internal/readme
go.opencensus.io/internal/tagencoding
go.opencensus.io/internal/testpb
go.opencensus.io/plugin/ocgrpc
go.opencensus.io/plugin/ochttp
go.opencensus.io/stats/internal
go.opencensus.io/stats/view
go.opencensus.io/trace/internal
go.opencensus.io/trace/propagation
go.opencensus.io/zpages/internal
gopkg.in/alecthomas/kingpin.v2
gopkg.in/bblfsh/client-go.v2
gopkg.in/bblfsh/client-go.v3
gopkg.in/bblfsh/sdk.v1
gopkg.in/bblfsh/sdk.v2
gopkg.in/go-playground/validator.v8
gopkg.in/inconshreveable/log15.v2
gopkg.in/mcuadros/go-defaults.v1
gopkg.in/mgo.v2/bson
gopkg.in/mgo.v2/dbtest
gopkg.in/mgo.v2/harness
gopkg.in/mgo.v2/internal
gopkg.in/mgo.v2/txn
gopkg.in/nullbio/null.v6
gopkg.in/olivere/elastic.v5
gopkg.in/russross/blackfriday.v2
gopkg.in/src-d/core-retrieval.v0
gopkg.in/src-d/enry.v1
gopkg.in/src-d/framework.v0
gopkg.in/src-d/go-billy-siva.v4
gopkg.in/src-d/go-billy.v4
gopkg.in/src-d/go-errors.v0
gopkg.in/src-d/go-errors.v1
gopkg.in/src-d/go-git.v4
gopkg.in/src-d/go-kallax.v1
gopkg.in/src-d/go-mysql-server.v0
gopkg.in/src-d/go-parse-utils.v1
gopkg.in/src-d/go-siva.v1
gopkg.in/src-d/go-vitess.v1
gopkg.in/toqueteos/substring.v1
gopkg.in/urfave/cli.v1
gopkg.in/vmihailenco/msgpack.v2
honnef.co/go/tools
sourcegraph.com/sqs/goreturns

erizocosmico commented 5 years ago

@eiso I have downloaded and tried with the dataset you provided but I could not reproduce the error. Perhaps some repos have different versions than what I got with go get?

erizocosmico commented 5 years ago

Gonna put the blocked tag until we find a reproduction case for this.

mcuadros commented 5 years ago

I guess is a life $GOPATH

erizocosmico commented 5 years ago

@eiso can you send me the command you use to start gitbase?

erizocosmico commented 5 years ago

ping @eiso

erizocosmico commented 5 years ago

Closing, since the issue cannot be reproduced. @eiso if you manage to find a reproducible case we can check feel free to ping us so we can reopen the issue and fix it. Most probable cause is starting gitbase on $GOPATH rather than $GOPATH/src and hitting a shallow repository.