Sync is not working with helm and jib

davidfernandezm commented 4 years ago

Expected behavior

Synced files should work

Actual behavior

No watching or syncing happening, no files going into the pod

Information

Skaffold version: 1.9.1
Operating system: Mac OS X Catalina 10.15.4
Contents of skaffold.yaml:

apiVersion: skaffold/v1
kind: Config
profiles:
  - name: qc
    build:
       artifacts:
        - image: skaffold-qc
          context: '.'
          sync:
            manual:
            - src: 'lombok.config'
              dest: /app
            - src: '**/*.config'
              dest: /app
            - src: 'KVP/heyo.txt'
              dest: /app/deploy
          jib: {}
       local: {}
    deploy:
      helm:
        releases:
          - name: "{{.USER}}-skaffold-qc"
            chartPath: "skaffold/qumucloud"
            values:
              image: skaffold-qc
            setValueTemplates:
              redisServerEndpoint: "dev-rds-euw1b-d01"
              redisServerPort: "6379"
        flags:
          upgrade:
            - --install
      statusCheckDeadlineSeconds: 240

Steps to reproduce the behavior

a clonable repository with the sample skaffold project: I cannot reproduce easily in the current project as it has too many folders. If nothing obvious shows up I'll try to extract
skaffold dev --filename skaffold.yml --profile qc --default-repo eu.gcr.io/qumu-dev --cleanup=false -v debug

The debug logs show up changes happening:

DEBU[0708] Change detected notify.Write: "/Users/dfernandez/development/repos/qumu-cloud-fork/qumu-cloud/skaffold/lombok.config"
DEBU[0708] Change detected notify.Write: "/Users/dfernandez/development/repos/qumu-cloud-fork/qumu-cloud/skaffold/lombok.config~"
DEBU[0708] Change detected notify.Create: "/Users/dfernandez/development/repos/qumu-cloud-fork/qumu-cloud/skaffold/lombok.config~"
DEBU[0708] Change detected notify.Remove: "/Users/dfernandez/development/repos/qumu-cloud-fork/qumu-cloud/skaffold/lombok.config~"
DEBU[0709] Using wrapper for gradlew: gradle
DEBU[0709] Found dependencies for jib-gradle artifact: [build.gradle build/libs/com/qumu/cloud/database/UpdateJobProcessorScriptsTableTask.class build/libs/commons-io-2.6.jar build/libs/liquibase-core-3.6.2.jar build/libs/logback-classic-1.2.3.jar build/libs/logback-core-1.2.3.jar build/libs/lombok-1.18.12.jar build/libs/lombok-1.18.4.jar build/libs/mariadb-java-client-2.4.3.jar build/libs/qumu-cloud.war build/libs/slf4j-api-1.7.25.jar build/libs/snakeyaml-1.18.jar database/liquibase/changesets/CLD-5865-pre.sql database/liquibase/changesets/DEV-9588.sql database/liquibase/changesets/DEV-9688.sql database/liquibase/changesets/QC-2323.sql database/liquibase/changesets/QC-2324.sql database/liquibase/changesets/QC-2644.sql database/liquibase/changese
....

And then more log with all the files detected by jib.

Is there anything obvious I am missing?

The structure is skaffold.yml and in the same folder lombok.config. Just seems the file is not being watched at all.

dgageot commented 4 years ago

ping @GoogleContainerTools/java-tools-build will jib know that lombok.config is a dependency?

loosebazooka commented 4 years ago

Thats interesting. How is the jib build configured? Can you run ./gradlew :<project>:_jibSkaffoldFilesV2 to see which files jib-skaffold is trying to watch? <-- lombok.config should be in here?

davidfernandezm commented 4 years ago

lombok.config is just an example of file that should be synced to the container. It could've been any txt or similar. I run a regular skaffold dev like the above, and the file never reaches the container, neither nothing gets printed in the logs. I scanned many issues around here.

I will try and test your command @loosebazooka and come back

davidfernandezm commented 4 years ago

When using JIB with Skaffold, can't I do then manual sync? all the files should be included as JIB dependencies so skaffold watches them? Could this be the issue?

davidfernandezm commented 4 years ago

@loosebazooka the output of that command is:

{"build":["/basefolder/build.gradle",
"/basefolder/settings.gradle",
"/basefolder/gradle.properties",
"/basefolder/modules/jms-google-pubsub/build.gradle",
"/basefolder/model/build.gradle",
"/basefolder/modules/common/build.gradle"],
"inputs":["/basefolder/src/main/resources",
"/basefolder/src/main/java",
"/basefolder/deploy/external-configuration",
"/basefolder/deploy/database",
"/basefolder/scripts/job-processor",
"/basefolder/database/liquibase",
"/basefolder/src/main/jib",
"/basefolder/modules/jms-google-pubsub/src/main/java",
"/basefolder/model/src/main/java",
"/basefolder/modules/common/src/main/java"],
"ignore":[]}

So looks like the sync, manual directive hasn't added that file to watch. I don't wanna track the files I've put in that directive through JIB, I just would like them sync-ed to the running container, not getting them to make the container rebuild.

loosebazooka commented 4 years ago

yeah unfortunately the way sync works, even in manual mode (@dgageot?) is that only files that are watched can be sync'd.

There are some workarounds here, but do require you to track the files in the build.gradle(not necessarily include them in the build thought)

The skaffold config block in the jib block allows you to kinda modify what information jib presents to skaffold.

so you could do something from (https://github.com/GoogleContainerTools/jib/tree/master/jib-gradle-plugin#skaffold-integration) like:

jib {
  skaffold {
    files {
      includes = ["lombok.config", "syncables/"]
    }
  }
}

and this wont affect your jib build (so these files aren't added to the container in the way that extraDirectories would), but will only modify the output of _jibSkaffoldFilesV2. It does affect how skaffold detects rebuilds though (which would be shortcircuited by your sync directives anyway?).

You could probably use some directory structures here to do what you want without having to change your config much?

NOTE: that sync block in the jib.skaffold config linked above is only applicable for sync:auto mode in skaffold

davidfernandezm commented 4 years ago

Thanks a lot @loosebazooka. I am very close to get something working properly, main issues atm with your workaround:

I can't use 'auto' in Skaffold sync, this is a WAR project and it's not support (obscure error message)
I am trying with 'manual' and it's fine for a single file. But with all the 'auto-xxx' (I press key to trigger, to make sure it is actioned while I watch) to false and for folders, no files get synced when changing them. If I add/delete files in the folder, they get synced (Syncing 1 file...). This could be due to few thousands of files in there, I just glimpsed once Syncing 2700 files...), could this be the case? As said, lombok.config is correctly picked up when pressing keys, but no *.class.

Current config:

# skaffold.yml
sync:
        manual:
        - src: 'lombok.config'
          dest: /app
        - src: 'KVP/**/*.class'
          dest: /usr/local/tomcat/webapps/ROOT

# build.gradle
...
jib {
...
skaffold {
    watch {
        includes = ["lombok.config", 'KVP/']
    }
}
}

loosebazooka commented 4 years ago

no files get synced when changing them

So in this case, do you mean you're changing the .java files? is the KVP/**/*.class path populated by some external build process that you are triggering?

I can't use 'auto' in Skaffold sync, this is a WAR project and it's not support (obscure error message)

Hrmm... maybe we should clean this up. Thanks for the feedback

davidfernandezm commented 4 years ago

Thanks for taking the time @loosebazooka,

KVP/**/*.class is indeed populated by IntelliJ IDEA compiling Java code.

Let me try to ellaborate the use case and see if I could use skaffold as a single tool for it:

I run skaffold dev so the overall JIB build is observed and triggered manually (not on source changes). So when a batch of changes (in java files, in significant amount of parts of the codebase) is completed, the whole container is built and deployed (manual trigger)
Once the full deployment is settled, further changes are done in the codebase, run tests, etc. These could be few lines (people debug this codebase a lot, making small changes to fix bugs). In this case, the preference is just syncing the compiled class files (by IntelliJ IDEA or Gradle compileJava). I've built a remote script that restarts Tomcat in the running container Skaffold deployed
So, if I get Skaffold to do both the whole JIB packaging, push and Helm deploy PLUS syncing of changed class files, config files... this is ideal because it's just one tool for everything. As you may have guessed, the initial build, package, push and deploy is slow (this is a big old app, takes nearly 2 minutes to complete a full build), so to get developers productive, syncing of compiled files is key (also for Frontend engineers, where the JS files get continuously synced).
I've successfully tried ksync for this whole amount of file syncing, and it works fine. But I'd rather stick with just skaffold as it seems it should still work
As a final note, I see the .class files landing in the folder, press a key in Skaffold interactive logs and nothing happens. If I change lombok.config, syncing happens, if I add/remove a file in KVP, syncing happens.

Thanks a lot for the dedication to the tool, it's great and a game changer for this team.

loosebazooka commented 4 years ago

It sounds like you're doing everything right. I think I need to try this out locally to maybe figure out what's going on. Developers on holiday for the next 4 days, so might be slow.

loosebazooka commented 4 years ago

So when recreating this I had to do a few things to get it to work in the way I think you want it to work. (I was using a springboot sample with springboot-devtools, but this should work for wars if configured similarly)

skaffold.yaml

build:
  artifacts:
  - image: gcr.io/loosebazooka-test/skaffold-jib
    jib:
      type: gradle
    sync:
      manual: 
        - src: "intellij-out/classes/**/*.class"
          dest: /app/classes/
          strip: "intellij-out/classes"

So when I do an intellij build, the intellij compiled classes are sent over to the remote container. I used strip to get the classes to go to the right place (which I'm not sure but maybe you need?)

jib (build.gradle)

jib {
  ...
  skaffold {
    watch {
      includes = ["intellij-out/classes"]
      excludes = ["src"]
    }
  }
}

I needed to ignore src because any changes to java files would trigger a full gradle jib rebuild. <-- perhaps this is the one thing that you need to add to your config?

davidfernandezm commented 4 years ago

I managed to get this working using your workaround, thanks a lot @loosebazooka. I am going to gather feedback from the team (specially frontend due to their workflows) and see if works for everyone. I will then close this issue. I think this is a nice workaround that would probably be outlined in the File Sync docs?

tejal29 commented 4 years ago

Thanks @davidfernandezm. We will work on adding docs

GoogleContainerTools / skaffold