search

versioneye / crawl_r

VersionEye crawlers implemented in Ruby.
https://www.versioneye.com
Other
10 stars 7 forks source link

Update GithubVersionCrawler with product version crawler that supports custom auth data. #41

Closed timgluz closed 7 years ago

timgluz commented 7 years ago

Hi,

this PR adds method GithubVersionCrawler.crawl_for_product, which pulls product's tags from Github API and updates product version with commit_sha and released_at.

I tried to re-use existing code, but it was difficult to get pagination to work without breaking current implementation. That's why i collected all the new functions into own module GithubVersionFetcher.

I also add worker and producer to get better interface for worker.

This commit also updated VersioneyeCore

Usage:

VersioneyeCore.new
prod_id = 'github.com/gorilla/mux' #expects that you already have such Go package in DB

#use Github tokens from running environments `Settings.json`
msg = GosearchVersionProducer.build_message(prod_id)

# use user Github token
msg = GosearchVersionProducer.build_message(prod_id, "reiz@versioneye.com")

# submit Octokit auth data directly
login_dt = {"login": "testveye", password: "check settings.json"}
msg = GosearchVersionProducer.build_message(prod_id, nil, login_dt)

# post the task
GosearchVersionProducer.new msg

# consume the task
VersionCrawlerWorker.new.work