Download File from URL.

[khoin]

Notes:

• https://github.com/cavaliercoder/grab

[khoin]

Hi @ctessum

We could allow people to specify the config file remotely and/or allow people to specify the shapefiles/datafiles in the config file remotely as well.

For now, I think we should only allow the shapefiles and datafiles to be URLs, because config files are usually lightweight and I don't see the reason for fetching a config file from afar. What do you think? [1]

I plan to use the package grab (linked in the main issue post) to download files as it streamlines some processes.

I just looked through the code of InMap, and I figured there are two possible places that I could extend for this fetching feature. One possibility is in cmd.go, another is in inmap.go.

Let me know what you think which direction I should take. [2] I think cmd.go is already a large file, but at the same time, inmap.go feels like it's at the core of the program and shouldn't hold instructions for side-stuff like downloading externally.

[ctessum]

Hi @khoin,

I think perhaps how things could be organized is that we could add another file to the inmaputil package called download.go, and in that file there could be a function called download that would:

1) take a string filepath as an argument 2) if the filepath is a remote location, download it into a temporary file, otherwise just return the filepath unmodified 3) If the downloaded temporary file is not a compressed file (i.e., tar.gz or zip), then return the path to the temporary file 3) If the downloaded temporary file is a compressed file, extract the contents in the temporary directory. 4) If there is only one extracted file, return the path to that file. 5) If there is more than one extracted file and one of them ends in the extension .shp, return the path to the file with the .shp extension. 6) Otherwise return the path to the compressed file.

Then, everywhere in the inmaputil package where a file path we can just wrap the file path in the download function. For instance, this line would change from

os.ExpandEnv(Cfg.GetString("VariableGridData")),

to

os.ExpandEnv(download(Cfg.GetString("VariableGridData"))),

Does that make sense?

You might be able to reuse some of the code for decompressing files from here.

[khoin]

Hi Chris,

That makes sense. I'll be looking for an extracting library since I expect we should want this to work on windows and shouldn't rely on Linux's built in extractor.

K On Mon, Feb 12, 2018 at 11:14 Chris Tessum notifications@github.com wrote:

Hi @khoin https://github.com/khoin,

I think perhaps how things could be organized is that we could add another file to the inmaputil package called download.go, and in that file there could be a function called download that would:

1. take a string filepath as an argument
2. if the filepath is a remote location, download it into a temporary file, otherwise just return the filepath unmodified
3. If the downloaded temporary file is not a compressed file (i.e., tar.gz or zip), then return the path to the temporary file
4. If the downloaded temporary file is a compressed file, extract the contents in the temporary directory.
5. If there is only one extracted file, return the path to that file.
6. If there is more than one extracted file and one of them ends in the extension .shp, return the path to the file with the .shp extension.
7. Otherwise return the path to the compressed file.

Then, everywhere in the inmaputil package where a file path we can just wrap the file path in the download function. For instance, this line https://github.com/spatialmodel/inmap/blob/master/inmaputil/cmd.go#L743 would change from

os.ExpandEnv(Cfg.GetString("VariableGridData")),

to

os.ExpandEnv(download(Cfg.GetString("VariableGridData"))),

Does that make sense?

You might be able to reuse some of the code for decompressing files from here https://github.com/ctessum/aep/blob/master/data/nei2014/download.go .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spatialmodel/inmap/issues/32#issuecomment-365030777, or mute the thread https://github.com/notifications/unsubscribe-auth/AEaAb-GndaDgrrWG9aDv6YUn9i8NDKGMks5tUI2ugaJpZM4SCk9c .

[ctessum]

The standard library has packages for compression. You can see how to use them here: https://github.com/ctessum/aep/blob/master/data/nei2014/download.go

On Mon, Feb 12, 2018 at 1:55 PM khoin notifications@github.com wrote:

Hi Chris,

That makes sense. I'll be looking for an extracting library since I expect we should want this to work on windows and shouldn't rely on Linux's built in extractor.

K On Mon, Feb 12, 2018 at 11:14 Chris Tessum notifications@github.com wrote:

Hi @khoin https://github.com/khoin,

I think perhaps how things could be organized is that we could add another file to the inmaputil package called download.go, and in that file there could be a function called download that would:

1. take a string filepath as an argument
2. if the filepath is a remote location, download it into a temporary file, otherwise just return the filepath unmodified
3. If the downloaded temporary file is not a compressed file (i.e., tar.gz or zip), then return the path to the temporary file
4. If the downloaded temporary file is a compressed file, extract the contents in the temporary directory.
5. If there is only one extracted file, return the path to that file.
6. If there is more than one extracted file and one of them ends in the extension .shp, return the path to the file with the .shp extension.
7. Otherwise return the path to the compressed file.

Then, everywhere in the inmaputil package where a file path we can just wrap the file path in the download function. For instance, this line <https://github.com/spatialmodel/inmap/blob/master/inmaputil/cmd.go#L743

would change from

os.ExpandEnv(Cfg.GetString("VariableGridData")),

to

os.ExpandEnv(download(Cfg.GetString("VariableGridData"))),

Does that make sense?

You might be able to reuse some of the code for decompressing files from here < https://github.com/ctessum/aep/blob/master/data/nei2014/download.go> .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/spatialmodel/inmap/issues/32#issuecomment-365030777 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AEaAb-GndaDgrrWG9aDv6YUn9i8NDKGMks5tUI2ugaJpZM4SCk9c

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spatialmodel/inmap/issues/32#issuecomment-365075649, or mute the thread https://github.com/notifications/unsubscribe-auth/AF6AAfffgnXsjEHUGwRVfPNz2utwyJXoks5tULM8gaJpZM4SCk9c .

[ctessum]

Fixed in #33