search

kevin-hanselman / dud

A lightweight CLI tool for versioning data alongside source code and building data pipelines.
https://kevin-hanselman.github.io/dud/
BSD 3-Clause "New" or "Revised" License
183 stars 8 forks source link

Verify remote authentication before attempting push and fetch #220

Open veriditin opened 1 month ago

veriditin commented 1 month ago

Dud seems to behave fairly inconsistently when authentication credentials for different remotes expire and/or are otherwise invalid.

With S3 as the remote, and an expired credential:

dud fetch - simply spins without any indication that something is wrong (Elapsed time goes up, nothing really happens), requires me to terminate the command (and then delete the lock file). dud pull - same as dud fetch (if cache is not up-to-date i guess) dud push - throws expiredToken exceptions, however repeats the command and only seems to halt after 2 minutes of trying:

pushing stage data.yaml
Gathering files      2                                                                                                  
2024/10/01 16:26:47 ERROR : S3 bucket <bucketname> path <dirname>: error reading destination root directory: ExpiredToken: The provided token has expired.
        status code: 400, <token info>
2024/10/01 16:26:47 ERROR : Attempt 1/3 failed with 1 errors and: ExpiredToken: The provided token has expired.
        status code: 400, <token info>
2024/10/01 16:27:23 ERROR : S3 bucket <bucketname> path <dirname>: error reading destination root directory: ExpiredToken: The provided token has expired.
        status code: 400,  <token info>
2024/10/01 16:27:23 ERROR : Attempt 2/3 failed with 1 errors and: ExpiredToken: The provided token has expired.
        status code: 400, <token info>
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      1m57.1s

My suggestion to add a lightweight check to the remote, before doing any significant work on a remote, and graciously exit the command if the remote is for whatever reason not available, indicating to the user that dud could not authenticate itself to the remote.

rclone ls seems to fit the bill for this lightweight check, if you don't make it recurse etc. - https://rclone.org/commands/rclone_ls/.

I don't have much experience developing in golang, but would this be a change you would be willing to receive?

kevin-hanselman commented 3 weeks ago

I don't have much experience developing in golang, but would this be a change you would be willing to receive?

Yes, I'd be happy to contribution on this! I think it's a good idea.