agunal / krantor

Automated download initiator for put.io. Fork of https://gitlab.com/klippz/krantor and https://gitlab.com/paulirish/krantor
5 stars 1 forks source link

hi from paulirish fork :) #1

Open paulirish opened 8 months ago

paulirish commented 8 months ago

i somehow missed your comment https://www.reddit.com/r/putdotio/comments/136u8r2/comment/jtb3g32/ but just came across it as i'm getting some unexpected 'context deadline exceeded' errors recently on upload. (still investigating)

A few unstructured thoughts...

  1. so happy you put your changes online!
  2. im not using it in docker, thus the lack of updates in dockerfile, but... i'm happy you updated that so it's a more complete package
  3. also.. i'm kinda surprised its fine in docker. part of my changes was removing radovskyb/watcher for fsnotify. watcher was polling every 100ms (geeeez!) vs fsnotify is based on inotify events and doesn't poll. (and uses way less cpu/power). but afterwards I read things about inotify not working within docker containers. i didnt look into it much so seemingly its a restriction in a smaller usecase.. as it seems to be fine for you. :) that's great news.
  4. i think this dep swap you did might fix my bug..... image especially since this.

i'm going to merge your fixes into my repo fwiw.

paulirish commented 8 months ago

question for you: have you encountered any problems since you did this work?

paulirish commented 8 months ago

Also.. looking at this commit what effect does using a separate thread have?

@@ -161,9 +166,9 @@ func watchFolder(client *putio.Client) {
                if !ok {
                    return
                }
                log.Println("event:", event) // Flip on for verbose logging
-               if event.Has(fsnotify.Create) { // AFAICT, I don't need to watch for move.
-                   prepareFile(event, client)
+               if event.Has(fsnotify.Create) {
+                   go prepareFile(event, client) // run in separate thread
                }
            case err, ok := <-w.Errors:
                if !ok {

and.. I guess you were seeing some race conditions where the file wasn't fully written?

@@ -119,6 +122,8 @@ func checkFileType(filename string) (string, error) {
 }

 func prepareFile(event fsnotify.Event, client *putio.Client) {
+   time.Sleep(100 * time.Millisecond) // wait for WRITE event(s) to finish
+
    var filepath string // todo, maybe remove?
    var err error
    var fileType string
paulirish commented 8 months ago

well the 'context deadline exceeded' bug was still around after applying all this and deps updates!

I think the putio servers are just too busy and timing out sometimes. I added some retry logic to handle that. but.. i haven't robustly tested it (as the servers got better and i couldn't repro anymore)!

anyhow. it now has everything from you, plus my new edits. hopefully its even better. cheers.

https://gitlab.com/paulirish/krantor