search

grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
965 stars 52 forks source link

reflow info fails, rebuild with fix also fails #141

Closed elimoss closed 2 years ago

elimoss commented 3 years ago

reflow info fails with reflow: taskdb: no provider for type taskdb.TaskDB

this seems to be similar to the problem encountered in #129, but the fix provided in that thread seems to be absent from the repo and the current release. I'd add it myself and rebuild, but when I try to do this I see

../../tool/cost.go:15:2: no required module provides package github.com/grailbio/base/cloud/spotfeed; to add it:
        go get github.com/grailbio/base/cloud/spotfeed
../../blob/s3blob/s3blob.go:21:2: no required module provides package github.com/grailbio/base/s3util; to add it:
        go get github.com/grailbio/base/s3util

and attempting to do this gives me

go get: module github.com/grailbio/base@upgrade found (v0.0.10), but does not contain package github.com/grailbio/base/cloud/spotfeed

and

go get: module github.com/grailbio/base@upgrade found (v0.0.10), but does not contain package github.com/grailbio/base/s3util

So I'm a bit stuck. I guess I need either a new release binary incorporating the fix from the previous issue (assuming that solves my problem, too) or a solution to my build issue.

Thanks in advance for any help you can provide!

swami-m commented 2 years ago

@elimoss Sorry for the delayed response. In order to use the reflow tools that rely on TaskDB (eg: reflow info), you need to have TaskDB configured in your reflow config. This isn't available as an option yet, but we are working on adding that option and it'll become available in a future release.

prb2 commented 2 years ago

but the fix provided in that thread seems to be absent from the repo and the current release

Just to clarify, the issue in #129 is not relevant for the problem you're running into. The issue there was reflow run was failing without TaskDB, even though it's not required. The fix provided in the comments was just a short term workaround, the proper fix for that issue was implemented in 3474fc2 and has been included since reflow1.3.1.

In your case, reflow info does require TaskDB as Swami stated, so that is the main hurdle.

swami-m commented 2 years ago

Sorry for the delay in making this work, but with the new 1.15.0 release, one can setup a taskdb if desired.