Cache downloaded file

[voytovichs]

Hi!

I searched web for solving a simple problem: download a file & reuse in further builds, I found quite nothing.

The plugin simply downloads file in a directory, it's easy to reuse that file if directory is the same, but this is not I'm looking for. This action could be done with a ant.get which is required no dependencies such as gradle-download-task plugin.

What ant.get can't do is a cache, i.e. download a file somewhere in .gradle directory and then always just copy it to output directory. And this would be a very nice feature to plugin like this. Also this feature coulnd make dependency configuration worthing for some people (at least me).

[michel-kraemer]

Hi @voytovichs!

I'm not sure if I understand what you're saying. Is this a question or a feature request? Could you please give me more details? Maybe an example?

I'm sure your problem is easy to solve. gradle-download-task is very flexible. I just need more information. Thanks.

Michel

[voytovichs]

Sorry for unclear description @michel-kraemer, I'm taking a second try:

First, it is a feature request. Gradle downloads configured dependencies once or at least until cache invalidation or deleting happened.

I have gradle test configuration where some files should be downloaded before tests started. Gradle downloads files every time I run build (download via ant.get task). Files are not changed from build to build and I have very natural desire - prevent new download and reuse files from previous build. Obviously I can check if there're necessary files and download them only if something is missing, but it seems that work should be done in another lawyer - at download task and by gradle, not by my build script.

So the use case should be clear at this point, the solution - cache downloaded files as a gradle build dependencies under .gradle directory and copy them instead of downloading.

Sergey

[plastiv]

@voytovichs have you tried onlyIfNewer,overwrite options?

[voytovichs]

@plastiv onlyIfNewer re-downloads only if file was modified on server. My file modifies locally and loses reusability

I can't see any help from overwrite either, file will be downloaded anyway

[plastiv]

You didn't mention you modifying file after downloading. Have you considered using Copy task (https://docs.gradle.org/current/dsl/org.gradle.api.tasks.Copy.html)?

• download task downloads file into downloadfiles/myfile
• copy task coping file into buildfiles/myfile
• buildfiles/myfile is modified during build
• next build copy task will detect that file is not the same and do copy again
• download task downloading file only once

[voytovichs]

@plastiv this is what I'm avoiding. Writing special task in my project makes it hardly (re)usable outside. Since I have a download task and default cache mechanism I don't want to invent a own cache in every particular download task usage and I don't want to define a special cache storage inside project. I want download tasks to do caching and use gradle default cache system.

I think that would be a very nice feature for a plugin that aims to do file download work :)

[michel-kraemer]

have you tried onlyIfNewer,overwrite options?

This should actually be the solution. If you set overwrite to false the plugin will never try to download the file again, no matter if it has been changed or not.

[michel-kraemer]

See here: https://github.com/michel-kraemer/gradle-download-task/blob/master/src/main/java/de/undercouch/gradle/tasks/download/DownloadAction.java#L135-L141

[voytovichs]

It doesn't fit my case: my downloaded file experiences transformations which make it non-reusable. Think of it as of a special prepared for tests database snapshot. Tests add, delete and update information in this database. If I rely on overwrite to false I wouldn't get fresh database, instead I'm going to use database which state is undefined (because previous tests run did whatever it wanted with data). So I need a new file, but I don't want it to be downloaded.

[michel-kraemer]

If you just want to create a new file that doesn't need to be download why do you use gradle-download-task in the first place?

I don't get it. You don't want to overwrite the file, but you also don't want to reuse it. I'm a bit puzzled. Sorry, for my ignorance. Maybe someone else can help?

[stigkj]

@voytovichs: you do need somewhere to cache the file. If that is the .gradle directory or somewhere else does not really matter, does it? Then the file must be copied to wherever you are working with it. This is exactly how one works with dependencies in gradle too. You never change a gradle dependency directly, instead it is copied somewhere before one can work on it.

[voytovichs]

@stigkj: you're right, it doesn't matter where I store caches as long as it one place for all of them. I'm agreed with everything you said.

@michel-kraemer you suggested to set overwrite to false. This satisfies me if the file wasn't changed. But if it was changed it has to be re-downloaded (I need it fresh). It would be very nice if your plugin gets the file "out of it's pocket", not from the internet.

[stigkj]

@voytovichs: If you have onlyIfNewer=true and overwrite=true the file will be cached and only be downloaded when it is changed on the server. Then you copy the file to a new directory where you can transform it. This should, as far as I can see, cover your use cases.

[voytovichs]

@stigkj: sorry for bothering you but can you please explain: let's assume onlyIfNewer=true and overwrite=true. File hasn't changed on server since last download and let's say it was removed in directory I passed as a dest parameter to downloadFile task. You say when I run downloadFile again the file is going to be put under dest directory? If answer is yes it is don't you know the location of temporary storage?

[michel-kraemer]

Thanks, @plastiv and @stigkj for the help.

@voytovichs I kind of start to understand what you are trying to achieve:

1. You want to download a file
2. Then modify the file locally
3. And download the file again when it has changed on the server. This will obviously overwrite your local changes.
4. Go back to step 2.

What I don't understand yet is why you are modifying the file locally in the first place. The local changes will be overwritten as soon as a new version of the file is downloaded.

The problem with this use case is that no plugin can somehow magically detect if a file has been changed on the server if you overwrite it locally. gradle-download-task relies on the last-modified timestamp of the local file and compares it to the one on the server (if onlyIfNewer is true). The only possibility that I see is that you manually save the last-modified timestamp before you modify the file (using File.lastModified()) and then set it back again (with File.setLastModified()), so that gradle-download-task doesn't see the local modification and sends the correct last-modified timestamp to the server (in the Last-Modified header) when it checks for remote changes the next time.

Does that help?

Michel

[michel-kraemer]

@voytovichs We haven't heard back from you since four weeks. Let us know if the tips and hints we gave you were actually helpful. If yes, please close the issue. If not, let us know what else we can do to help you. Thanks.

[michel-kraemer]

There was no activity in this thread for more than seven weeks now. I assume the issue has been resolved and will close it. @voytovichs if the problem still exists feel free to reopen the issue. We'd be more than happy to help you.