search

datatogether / task_mgmt

Service for managing & executing archiving tasks written in Golang
https://task-mgmt.archivers.space
GNU Affero General Public License v3.0
2 stars 1 forks source link

Project Open Data & Sciencebase Catalog Archiving support #11

Closed b5 closed 6 years ago

b5 commented 6 years ago

Ugh sorry I've left this open for a while, this is the code I've been using to archive POD data.json catalogs and sciencebase data catalogs. Both are defined as new types of tasks, and both leverage goroutines to crawl the catalogs in parallel, which makes the archiving process much more manageable time wise. I'd like to get this merged before the sprint if possible!

codecov[bot] commented 6 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ae20a8c). Click here to learn what that means. The diff coverage is 2.43%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #11   +/-   ##
=========================================
  Coverage          ?   16.55%           
=========================================
  Files             ?       11           
  Lines             ?      586           
  Branches          ?        0           
=========================================
  Hits              ?       97           
  Misses            ?      478           
  Partials          ?       11
Impacted Files Coverage Δ
redis.go 0% <ø> (ø)
rpc.go 0% <ø> (ø)
server.go 8% <0%> (ø)
accept_tasks.go 0% <0%> (ø)
handlers.go 0% <0%> (ø)
middleware.go 16.66% <0%> (ø)
config.go 56.52% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ae20a8c...6548927. Read the comment docs.

b5 commented 6 years ago

I've also hijacked this PR to fulfill the Sprint prep checklist. This PR should: * Update repo to new package names (archive -> core, etc.)

titaniumbones commented 6 years ago

Not ignoring this! trying to find time to do at least a minimally responsible review, but feel free to merge if you get tired of waiting!

b5 notifications@github.com writes:

@b5 requested your review on: datatogether/task_mgmt#11 Project Open Data & Sciencebase Catalog Archiving support.

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHub, or mute the thread.

--

dcwalk commented 6 years ago

@b5 -- could you talk me through why are we moving to readme_prev? Are we eventually going to merge content back in?

dcwalk commented 6 years ago

RE: codecov error, here is the documentation: https://docs.codecov.io/docs/comparing-commits

I think we can just merge this into master, future PRs that branch off that should be able to have a comparison as the code coverage lines will be added to circle ci config...?

dcwalk commented 6 years ago

This PR should address #12 and https://github.com/datatogether/roadmap/issues/53