fabric8-analytics / fabric8-analytics-jobs

fabric8-analytics periodic and query-specific jobs
Apache License 2.0
5 stars 30 forks source link

Golang package containing github.com/A/B/C... fails #208

Closed tuxdna closed 6 years ago

tuxdna commented 6 years ago

Any golang package contain more than three components fails because latest commit information for that package couldn't be found. Below is a list of packages ( from top 400 popular golang packages ), which fail to schedule:

     11:2017-11-08 04:40:13,900 - Couldn't get latest commit for golang.org/x/net/context
     22:2017-11-08 04:40:17,061 - Couldn't get latest commit for gopkg.in/yaml.v2
     27:2017-11-08 04:40:18,082 - Couldn't get latest commit for gopkg.in/mgo.v2
     29:2017-11-08 04:40:18,082 - Couldn't get latest commit for gopkg.in/mgo.v2/bson
     34:2017-11-08 04:40:18,986 - Couldn't get latest commit for github.com/garyburd/redigo/redis
     42:2017-11-08 04:40:21,292 - Couldn't get latest commit for golang.org/x/oauth2
     47:2017-11-08 04:40:22,555 - Couldn't get latest commit for github.com/golang/protobuf/proto
     61:2017-11-08 04:40:25,889 - Couldn't get latest commit for github.com/aws/aws-sdk-go/aws
     72:2017-11-08 04:40:28,778 - Couldn't get latest commit for golang.org/x/net/websocket
     77:2017-11-08 04:40:29,773 - Couldn't get latest commit for golang.org/x/crypto/bcrypt
     85:2017-11-08 04:40:31,383 - Couldn't get latest commit for labix.org/v2/mgo
     90:2017-11-08 04:40:32,288 - Couldn't get latest commit for labix.org/v2/mgo/bson
     92:2017-11-08 04:40:32,288 - Couldn't get latest commit for golang.org/x/net/html
     94:2017-11-08 04:40:32,348 - Couldn't get latest commit for github.com/aws/aws-sdk-go/aws/session
    105:2017-11-08 04:40:35,394 - Couldn't get latest commit for golang.org/x/crypto/ssh
    116:2017-11-08 04:40:38,614 - Couldn't get latest commit for github.com/google/go-github/github
    124:2017-11-08 04:40:40,388 - Couldn't get latest commit for golang.org/x/crypto/ssh/terminal
    126:2017-11-08 04:40:40,388 - Couldn't get latest commit for google.golang.org/grpc
    137:2017-11-08 04:40:42,685 - Couldn't get latest commit for golang.org/x/oauth2/google
    160:2017-11-08 04:40:49,711 - Couldn't get latest commit for github.com/zenazn/goji/web
    162:2017-11-08 04:40:49,753 - Couldn't get latest commit for github.com/davecgh/go-spew/spew
    176:2017-11-08 04:40:52,674 - Couldn't get latest commit for github.com/aws/aws-sdk-go/aws/credentials
    181:2017-11-08 04:40:54,318 - Couldn't get latest commit for github.com/coreos/go-etcd/etcd
    186:2017-11-08 04:40:55,892 - Couldn't get latest commit for gopkg.in/fsnotify.v1
    188:2017-11-08 04:40:55,983 - Couldn't get latest commit for github.com/aws/aws-sdk-go/service/s3
    193:2017-11-08 04:40:57,015 - Couldn't get latest commit for github.com/astaxie/beego/orm
    198:2017-11-08 04:40:58,067 - Couldn't get latest commit for gopkg.in/redis.v3
    203:2017-11-08 04:40:59,198 - Couldn't get latest commit for gopkg.in/alecthomas/kingpin.v2
    205:2017-11-08 04:40:59,284 - Couldn't get latest commit for github.com/syndtr/goleveldb/leveldb
    207:2017-11-08 04:40:59,327 - Couldn't get latest commit for github.com/prometheus/client_golang/prometheus
    212:2017-11-08 04:41:00,168 - Couldn't get latest commit for google.golang.org/appengine
    214:2017-11-08 04:41:00,169 - Couldn't get latest commit for gopkg.in/yaml.v1
    219:2017-11-08 04:41:01,120 - Couldn't get latest commit for github.com/hashicorp/consul/api
    221:2017-11-08 04:41:01,177 - Couldn't get latest commit for github.com/coreos/etcd/client
    235:2017-11-08 04:41:05,179 - Couldn't get latest commit for github.com/aws/aws-sdk-go/service/ec2
    237:2017-11-08 04:41:05,226 - Couldn't get latest commit for github.com/labstack/echo/middleware
    245:2017-11-08 04:41:07,406 - Couldn't get latest commit for github.com/gogo/protobuf/proto
    253:2017-11-08 04:41:09,119 - Couldn't get latest commit for github.com/ugorji/go/codec
    258:2017-11-08 04:41:10,091 - Couldn't get latest commit for golang.org/x/text/transform
    272:2017-11-08 04:41:14,319 - Couldn't get latest commit for github.com/ant0ine/go-json-rest/rest
    280:2017-11-08 04:41:16,581 - Couldn't get latest commit for github.com/stretchr/testify/mock
    285:2017-11-08 04:41:17,641 - Couldn't get latest commit for github.com/bradfitz/gomemcache/memcache
    299:2017-11-08 04:41:21,457 - Couldn't get latest commit for google.golang.org/appengine/datastore
    301:2017-11-08 04:41:21,500 - Couldn't get latest commit for github.com/aws/aws-sdk-go/aws/awserr
    303:2017-11-08 04:41:21,553 - Couldn't get latest commit for github.com/gopherjs/gopherjs/js
    311:2017-11-08 04:41:23,769 - Couldn't get latest commit for google.golang.org/appengine/log
    313:2017-11-08 04:41:23,815 - Couldn't get latest commit for github.com/samuel/go-zookeeper/zk
    321:2017-11-08 04:41:25,671 - Couldn't get latest commit for launchpad.net/goyaml
    323:2017-11-08 04:41:25,671 - Couldn't get latest commit for golang.org/x/sys/unix
    328:2017-11-08 04:41:26,486 - Couldn't get latest commit for github.com/google/go-querystring/query
    339:2017-11-08 04:41:30,513 - Couldn't get latest commit for github.com/mitchellh/goamz/aws
    344:2017-11-08 04:41:31,601 - Couldn't get latest commit for github.com/codegangsta/martini-contrib/render
    346:2017-11-08 04:41:31,601 - Couldn't get latest commit for golang.org/x/crypto/scrypt
    348:2017-11-08 04:41:31,642 - Couldn't get latest commit for github.com/influxdb/influxdb/client
    350:2017-11-08 04:41:31,686 - Couldn't get latest commit for github.com/labstack/echo/engine/standard
    352:2017-11-08 04:41:31,686 - Couldn't get latest commit for golang.org/x/crypto/pbkdf2
    354:2017-11-08 04:41:31,729 - Couldn't get latest commit for github.com/skratchdot/open-golang/open
    365:2017-11-08 04:41:34,358 - Couldn't get latest commit for golang.org/x/net/html/atom
    379:2017-11-08 04:41:39,079 - Couldn't get latest commit for github.com/astaxie/beego/context
    387:2017-11-08 04:41:40,965 - Couldn't get latest commit for github.com/docker/docker/pkg/term
    392:2017-11-08 04:41:42,149 - Couldn't get latest commit for github.com/stretchr/testify/assert
    397:2017-11-08 04:41:43,079 - Couldn't get latest commit for google.golang.org/appengine/urlfetch
    399:2017-11-08 04:41:43,079 - Couldn't get latest commit for golang.org/x/crypto/openpgp
    401:2017-11-08 04:41:43,126 - Couldn't get latest commit for github.com/influxdata/influxdb/client/v2
    406:2017-11-08 04:41:44,158 - Couldn't get latest commit for gopkg.in/gcfg.v1
    417:2017-11-08 04:41:47,004 - Couldn't get latest commit for github.com/revel/revel/testing
    425:2017-11-08 04:41:48,906 - Couldn't get latest commit for github.com/go-kit/kit/log
    427:2017-11-08 04:41:48,948 - Couldn't get latest commit for github.com/fzzy/radix/redis
    441:2017-11-08 04:41:51,167 - Couldn't get latest commit for github.com/docker/docker/pkg/archive
    443:2017-11-08 04:41:51,231 - Couldn't get latest commit for github.com/go-gl/glfw/v3.1/glfw
    451:2017-11-08 04:41:53,683 - Couldn't get latest commit for golang.org/x/net/html/charset
    465:2017-11-08 04:41:58,297 - Couldn't get latest commit for golang.org/x/crypto/nacl/secretbox
    467:2017-11-08 04:41:58,387 - Couldn't get latest commit for github.com/mitchellh/goamz/s3
    469:2017-11-08 04:41:58,388 - Couldn't get latest commit for golang.org/x/net/context/ctxhttp
    471:2017-11-08 04:41:58,434 - Couldn't get latest commit for github.com/veandco/go-sdl2/sdl
    476:2017-11-08 04:41:59,597 - Couldn't get latest commit for github.com/syndtr/goleveldb/leveldb/util
    481:2017-11-08 04:42:00,560 - Couldn't get latest commit for golang.org/x/text/unicode/norm
    483:2017-11-08 04:42:00,560 - Couldn't get latest commit for golang.org/x/crypto/ssh/agent
    485:2017-11-08 04:42:00,561 - Couldn't get latest commit for gopkg.in/inconshreveable/log15.v2
    490:2017-11-08 04:42:01,824 - Couldn't get latest commit for github.com/golang/groupcache/lru
    495:2017-11-08 04:42:03,184 - Couldn't get latest commit for launchpad.net/goamz/aws
    497:2017-11-08 04:42:03,226 - Couldn't get latest commit for github.com/garyburd/go-oauth/oauth
    499:2017-11-08 04:42:03,298 - Couldn't get latest commit for github.com/astaxie/beego/validation
    507:2017-11-08 04:42:05,856 - Couldn't get latest commit for golang.org/x/tools/go/loader
    509:2017-11-08 04:42:05,921 - Couldn't get latest commit for github.com/google/gopacket/layers
    523:2017-11-08 04:42:10,428 - Couldn't get latest commit for github.com/dotcloud/docker/utils
    528:2017-11-08 04:42:11,486 - Couldn't get latest commit for github.com/syndtr/goleveldb/leveldb/opt
    530:2017-11-08 04:42:11,487 - Couldn't get latest commit for google.golang.org/grpc/codes
    547:2017-11-08 04:42:16,487 - Couldn't get latest commit for gopkg.in/gorp.v1
    549:2017-11-08 04:42:16,537 - Couldn't get latest commit for github.com/docker/docker/registry
    551:2017-11-08 04:42:16,546 - Couldn't get latest commit for golang.org/x/net/http2
    553:2017-11-08 04:42:16,595 - Couldn't get latest commit for github.com/tensorflow/tensorflow/tensorflow/go/genop/internal
    555:2017-11-08 04:42:16,652 - Couldn't get latest commit for github.com/tensorflow/tensorflow/tensorflow/go
    557:2017-11-08 04:42:16,653 - Couldn't get latest commit for k8s.io/kubernetes/pkg/api
    559:2017-11-08 04:42:16,726 - Couldn't get latest commit for github.com/fluffle/goirc/client
    564:2017-11-08 04:42:17,433 - Couldn't get latest commit for github.com/docker/docker/pkg/ioutils
    566:2017-11-08 04:42:17,433 - Couldn't get latest commit for gopkg.in/alecthomas/kingpin.v1
    568:2017-11-08 04:42:17,495 - Couldn't get latest commit for github.com/crowdmob/goamz/aws
    570:2017-11-08 04:42:17,553 - Couldn't get latest commit for github.com/docker/docker/pkg/parsers
    572:2017-11-08 04:42:17,594 - Couldn't get latest commit for github.com/dotcloud/docker/registry
    580:2017-11-08 04:42:19,724 - Couldn't get latest commit for github.com/docker/docker/opts
    582:2017-11-08 04:42:19,725 - Couldn't get latest commit for golang.org/x/crypto/nacl/box
    587:2017-11-08 04:42:20,285 - Couldn't get latest commit for google.golang.org/appengine/memcache
    589:2017-11-08 04:42:20,331 - Couldn't get latest commit for github.com/docker/engine-api/types
    594:2017-11-08 04:42:21,925 - Couldn't get latest commit for github.com/docker/docker/pkg/jsonmessage
    608:2017-11-08 04:42:26,649 - Couldn't get latest commit for github.com/hashicorp/terraform/helper/schema
    610:2017-11-08 04:42:26,696 - Couldn't get latest commit for github.com/google/gopacket/pcap
    612:2017-11-08 04:42:26,742 - Couldn't get latest commit for github.com/go-kit/kit/endpoint
    614:2017-11-08 04:42:26,742 - Couldn't get latest commit for gopkg.in/ini.v1
    616:2017-11-08 04:42:26,789 - Couldn't get latest commit for github.com/docker/docker/pkg/stdcopy
    621:2017-11-08 04:42:27,801 - Couldn't get latest commit for bazil.org/fuse
    629:2017-11-08 04:42:30,282 - Couldn't get latest commit for github.com/docker/docker/pkg/fileutils
    634:2017-11-08 04:42:31,221 - Couldn't get latest commit for github.com/zenazn/goji/web/middleware
    636:2017-11-08 04:42:31,267 - Couldn't get latest commit for github.com/hashicorp/terraform/terraform
    638:2017-11-08 04:42:31,267 - Couldn't get latest commit for bitbucket.org/kardianos/osext
    640:2017-11-08 04:42:31,310 - Couldn't get latest commit for github.com/aws/aws-sdk-go/aws/ec2metadata
    642:2017-11-08 04:42:31,311 - Couldn't get latest commit for golang.org/x/net/ipv4
    644:2017-11-08 04:42:31,311 - Couldn't get latest commit for gopkg.in/tylerb/graceful.v1
    646:2017-11-08 04:42:31,353 - Couldn't get latest commit for github.com/aws/aws-sdk-go/service/s3/s3manager
    648:2017-11-08 04:42:31,398 - Couldn't get latest commit for github.com/docker/docker/image
    650:2017-11-08 04:42:31,440 - Couldn't get latest commit for github.com/ziutek/mymysql/godrv
    655:2017-11-08 04:42:32,607 - Couldn't get latest commit for github.com/docker/docker/daemon/graphdriver
    657:2017-11-08 04:42:32,663 - Couldn't get latest commit for github.com/docker/engine-api/client
    659:2017-11-08 04:42:32,719 - Couldn't get latest commit for github.com/docker/docker/pkg/tarsum
    661:2017-11-08 04:42:32,768 - Couldn't get latest commit for github.com/go-kit/kit/transport/http
    663:2017-11-08 04:42:32,820 - Couldn't get latest commit for github.com/docker/docker/pkg/parsers/kernel
    668:2017-11-08 04:42:33,799 - Couldn't get latest commit for bazil.org/fuse/fs
    670:2017-11-08 04:42:33,884 - Couldn't get latest commit for github.com/docker/docker/pkg/symlink
    672:2017-11-08 04:42:33,884 - Couldn't get latest commit for launchpad.net/goamz/s3
    674:2017-11-08 04:42:33,885 - Couldn't get latest commit for golang.org/x/tools/go/types
    682:2017-11-08 04:42:35,812 - Couldn't get latest commit for github.com/docker/docker/pkg/mount
    684:2017-11-08 04:42:35,858 - Couldn't get latest commit for github.com/docker/docker/utils
    686:2017-11-08 04:42:35,900 - Couldn't get latest commit for github.com/GoogleCloudPlatform/kubernetes/pkg/api
    688:2017-11-08 04:42:35,943 - Couldn't get latest commit for github.com/docker/docker/pkg/system
    693:2017-11-08 04:42:37,086 - Couldn't get latest commit for github.com/docker/docker/pkg/namesgenerator
    695:2017-11-08 04:42:37,127 - Couldn't get latest commit for github.com/rwcarlsen/goexif/exif
    697:2017-11-08 04:42:37,128 - Couldn't get latest commit for golang.org/x/tools/imports
    699:2017-11-08 04:42:37,166 - Couldn't get latest commit for github.com/go-gl/mathgl/mgl32
    704:2017-11-08 04:42:37,823 - Couldn't get latest commit for github.com/astaxie/beego/logs
    706:2017-11-08 04:42:37,823 - Couldn't get latest commit for golang.org/x/net/publicsuffix
    711:2017-11-08 04:42:39,111 - Couldn't get latest commit for github.com/docker/docker/api/client
    716:2017-11-08 04:42:39,904 - Couldn't get latest commit for github.com/golang/mock/gomock
    718:2017-11-08 04:42:39,951 - Couldn't get latest commit for github.com/docker/docker/pkg/signal
    720:2017-11-08 04:42:39,951 - Couldn't get latest commit for google.golang.org/grpc/credentials
    725:2017-11-08 04:42:41,022 - Couldn't get latest commit for github.com/docker/docker/pkg/truncindex
    727:2017-11-08 04:42:41,069 - Couldn't get latest commit for github.com/hashicorp/terraform/plugin
    732:2017-11-08 04:42:41,879 - Couldn't get latest commit for golang.org/x/net/proxy
    734:2017-11-08 04:42:41,923 - Couldn't get latest commit for github.com/docker/docker/api
    736:2017-11-08 04:42:42,005 - Couldn't get latest commit for github.com/wsxiaoys/terminal/color
    741:2017-11-08 04:42:43,301 - Couldn't get latest commit for github.com/aws/aws-sdk-go/service/dynamodb
    743:2017-11-08 04:42:43,342 - Couldn't get latest commit for github.com/shirou/gopsutil/mem
    745:2017-11-08 04:42:43,343 - Couldn't get latest commit for google.golang.org/cloud
    747:2017-11-08 04:42:43,385 - Couldn't get latest commit for github.com/gonum/matrix/mat64
    755:2017-11-08 04:42:45,380 - Couldn't get latest commit for github.com/golang/freetype/truetype
    757:2017-11-08 04:42:45,421 - Couldn't get latest commit for github.com/mattbaird/elastigo/lib
    759:2017-11-08 04:42:45,421 - Couldn't get latest commit for k8s.io/kubernetes/pkg/client/unversioned
    761:2017-11-08 04:42:45,464 - Couldn't get latest commit for github.com/docker/docker/pkg/httputils
    763:2017-11-08 04:42:45,465 - Couldn't get latest commit for google.golang.org/grpc/grpclog
    765:2017-11-08 04:42:45,465 - Couldn't get latest commit for google.golang.org/grpc/metadata
    767:2017-11-08 04:42:45,465 - Couldn't get latest commit for git.apache.org/thrift.git/lib/go/thrift
    772:2017-11-08 04:42:46,327 - Couldn't get latest commit for github.com/docker/docker/pkg/parsers/operatingsystem
    774:2017-11-08 04:42:46,328 - Couldn't get latest commit for google.golang.org/cloud/storage
    776:2017-11-08 04:42:46,386 - Couldn't get latest commit for github.com/docker/docker/pkg/promise
    781:2017-11-08 04:42:47,127 - Couldn't get latest commit for github.com/docker/docker/runconfig
    783:2017-11-08 04:42:47,185 - Couldn't get latest commit for github.com/docker/docker/pkg/chrootarchive
    785:2017-11-08 04:42:47,229 - Couldn't get latest commit for github.com/docker/docker/pkg/sysinfo
    811:2017-11-08 04:42:54,638 - Couldn't get latest commit for github.com/docker/docker/pkg/graphdb
    813:2017-11-08 04:42:54,639 - Couldn't get latest commit for google.golang.org/api/googleapi
    815:2017-11-08 04:42:54,709 - Couldn't get latest commit for github.com/ziutek/mymysql/mysql
    820:2017-11-08 04:42:55,918 - Couldn't get latest commit for github.com/awslabs/aws-sdk-go/aws
    825:2017-11-08 04:42:56,741 - Couldn't get latest commit for github.com/docker/docker/pkg/urlutil
    827:2017-11-08 04:42:56,804 - Couldn't get latest commit for github.com/astaxie/beego/session
    835:2017-11-08 04:42:59,787 - Couldn't get latest commit for github.com/mitchellh/packer/packer
    837:2017-11-08 04:42:59,843 - Couldn't get latest commit for github.com/pquerna/ffjson/ffjson
    839:2017-11-08 04:42:59,900 - Couldn't get latest commit for github.com/goamz/goamz/aws
    844:2017-11-08 04:43:00,822 - Couldn't get latest commit for github.com/aws/aws-sdk-go/service/sqs
    846:2017-11-08 04:43:00,865 - Couldn't get latest commit for github.com/astaxie/beego/config
    848:2017-11-08 04:43:00,866 - Couldn't get latest commit for golang.org/x/crypto/openpgp/armor
    856:2017-11-08 04:43:03,890 - Couldn't get latest commit for github.com/docker/machine/libmachine/drivers
    861:2017-11-08 04:43:05,598 - Couldn't get latest commit for gopkg.in/olivere/elastic.v3
    863:2017-11-08 04:43:05,598 - Couldn't get latest commit for git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.golang.git
    865:2017-11-08 04:43:05,672 - Couldn't get latest commit for github.com/docker/docker/cliconfig
    867:2017-11-08 04:43:05,714 - Couldn't get latest commit for github.com/docker/docker/pkg/pools
    869:2017-11-08 04:43:05,715 - Couldn't get latest commit for golang.org/x/text/language
    871:2017-11-08 04:43:05,761 - Couldn't get latest commit for github.com/facebookgo/grace/gracehttp
    879:2017-11-08 04:43:08,407 - Couldn't get latest commit for github.com/mitchellh/packer/packer/plugin
    881:2017-11-08 04:43:08,460 - Couldn't get latest commit for github.com/docker/machine/libmachine/state
    883:2017-11-08 04:43:08,512 - Couldn't get latest commit for github.com/astaxie/beego/cache
    885:2017-11-08 04:43:08,513 - Couldn't get latest commit for golang.org/x/image/math/fixed
    887:2017-11-08 04:43:08,556 - Couldn't get latest commit for github.com/BurntSushi/xgb/xproto
    892:2017-11-08 04:43:09,947 - Couldn't get latest commit for github.com/dotcloud/docker/pkg/term
    894:2017-11-08 04:43:09,996 - Couldn't get latest commit for github.com/hybridgroup/gobot/platforms/gpio
    908:2017-11-08 04:43:13,494 - Couldn't get latest commit for github.com/zenazn/goji/graceful
    910:2017-11-08 04:43:13,537 - Couldn't get latest commit for github.com/docker/docker/daemon/logger
    915:2017-11-08 04:43:14,579 - Couldn't get latest commit for github.com/hashicorp/vault/api
    920:2017-11-08 04:43:15,996 - Couldn't get latest commit for gopkg.in/gomail.v2
    922:2017-11-08 04:43:16,081 - Couldn't get latest commit for github.com/docker/docker/pkg/reexec
    924:2017-11-08 04:43:16,125 - Couldn't get latest commit for github.com/docker/docker/pkg/streamformatter
    929:2017-11-08 04:43:17,298 - Couldn't get latest commit for golang.org/x/oauth2/jwt
    931:2017-11-08 04:43:17,377 - Couldn't get latest commit for github.com/docker/docker/daemon/logger/jsonfilelog
    939:2017-11-08 04:43:19,693 - Couldn't get latest commit for golang.org/x/text/encoding/charmap
    941:2017-11-08 04:43:19,693 - Couldn't get latest commit for golang.org/x/crypto/openpgp/packet
    943:2017-11-08 04:43:19,693 - Couldn't get latest commit for golang.org/x/crypto/sha3
    945:2017-11-08 04:43:19,796 - Couldn't get latest commit for github.com/GoogleCloudPlatform/kubernetes/pkg/labels
    947:2017-11-08 04:43:19,839 - Couldn't get latest commit for github.com/mitchellh/packer/common
    949:2017-11-08 04:43:19,880 - Couldn't get latest commit for github.com/mozilla-services/heka/pipeline
    951:2017-11-08 04:43:19,925 - Couldn't get latest commit for github.com/emicklei/go-restful/swagger
    953:2017-11-08 04:43:19,967 - Couldn't get latest commit for github.com/docker/machine/libmachine/mcnflag
    955:2017-11-08 04:43:20,008 - Couldn't get latest commit for github.com/docker/docker/daemon
    957:2017-11-08 04:43:20,009 - Couldn't get latest commit for gopkg.in/vmihailenco/msgpack.v2
    959:2017-11-08 04:43:20,058 - Couldn't get latest commit for github.com/docker/docker/api/types
    964:2017-11-08 04:43:20,676 - Couldn't get latest commit for golang.org/x/text/encoding
    975:2017-11-08 04:43:24,511 - Couldn't get latest commit for github.com/docker/machine/libmachine/log
    977:2017-11-08 04:43:24,555 - Couldn't get latest commit for github.com/docker/docker/pkg/stringid
    979:2017-11-08 04:43:24,599 - Couldn't get latest commit for github.com/kylelemons/go-gypsy/yaml
    981:2017-11-08 04:43:24,644 - Couldn't get latest commit for github.com/gogits/gogs/modules/setting
    983:2017-11-08 04:43:24,688 - Couldn't get latest commit for github.com/codegangsta/martini-contrib/binding
    991:2017-11-08 04:43:27,177 - Couldn't get latest commit for github.com/shirou/gopsutil/cpu
    993:2017-11-08 04:43:27,221 - Couldn't get latest commit for github.com/docker/docker/pkg/tailfile
    995:2017-11-08 04:43:27,221 - Couldn't get latest commit for gopkg.in/qml.v1
tuxdna commented 6 years ago

With the above data in file golang-400-job.txt, and using following script find_failing.py:

import re
import itertools

data = []
for line in open("golang-400-job.txt").readlines():
    s = line.strip()
    m = re.match(".*Couldn't get latest commit for (\S+).*", s)
    if m:
        data.append(m.groups()[0])

data = sorted(data)
grouped_repos = [(k, list(map(lambda x: x[1:], v)))
                 for k, v in itertools.groupby(map(lambda x: x.split("/"), data), lambda x: x[0])]

grouped_repos = sorted(grouped_repos, key=lambda x: -len(x[1]))

for k, v in grouped_repos:
    print(k, ": No of packages", len(v))
$ python find_failing.py 
github.com : No of packages 143
golang.org : No of packages 35
gopkg.in : No of packages 17
google.golang.org : No of packages 13
launchpad.net : No of packages 3
bazil.org : No of packages 2
k8s.io : No of packages 2
labix.org : No of packages 2
bitbucket.org : No of packages 1
git.apache.org : No of packages 1
git.eclipse.org : No of packages 1
tuxdna commented 6 years ago

Failure happens due to https://github.com/fabric8-analytics/fabric8-analytics-jobs/blob/37fa4aaeb0d7a0ce56ee2f337c5731e48a9de741/f8a_jobs/handlers/golang_popular_analyses.py#L14

In this case for example github.com/garyburd/redigo/redis will result into

>>> url = 'https://{p}/commits/master'.format(p='github.com/garyburd/redigo/redis')

https://github.com/garyburd/redigo/redis/commits/master

The above url returns 404 error.

We should instead pick the first three components of package name to for the url github.com/garyburd/redigo/

In [33]: package='github.com/garyburd/redigo'

In [34]: %paste
def _get_latest_commit(package):
    if package.startswith('github.com'):
        url = 'https://{p}/commits/master'.format(p=package)
        response = requests.get(url)
        if response.status_code == 200:
            page = BeautifulSoup(response.text, 'html.parser')
            commit_links = page.find_all(class_='commit-links-group BtnGroup')
            if commit_links:
                commit_tag = commit_links[0].find_next('a')
                if commit_tag:
                    link = commit_tag.get('href', '')
                    if link and '/' in link:
                        return link.split('/')[-1]
    return None
## -- End pasted text --

In [35]: _get_latest_commit(package)
Out[35]: '47dc60e71eed504e3ef8e77ee3c6fe720f3be57f'
tuxdna commented 6 years ago

Closing via https://github.com/fabric8-analytics/fabric8-analytics-jobs/pull/209