Directory Structure for Files

[benjiao]

From @benjiao on November 25, 2015 9:23

Files include:

• Bundles (compressed)
• Individual TIFs
• Thumbnails
• Tiles

Here's my proposed structure:

.
+-- images
|   +-- landsat
|       +-- LC81180542015327LGN00
|           +-- LC81180542015327LGN00_THUMB
|           +-- LC81180542015327LGN00_B1
|           +-- LC81180542015327LGN00_B2
|           +-- LC81180542015327LGN00_B3
|       +-- LC81180542015327LGN01
|           +-- LC81180542015327LGN01_THUMB
|           +-- LC81180542015327LGN01_B1
|           +-- LC81180542015327LGN01_B2
|   +-- diwata1
|       +-- D1C81180542015327LGN00
|           +-- D1C81180542015327LGN00_THUMB
|           +-- D1C81180542015327LGN00_B1
|           +-- D1C81180542015327LGN00_B2
|       +-- D1C81180542015327LGN01
|           +-- D1C81180542015327LGN01_THUMB
|           +-- D1C81180542015327LGN01_B1
|           +-- D1C81180542015327LGN01_B2
|           +-- D1C81180542015327LGN01_B4

Landsat API and Libra will depend on the _<band-id> suffix to tell whether a specific band or data product is available for that scene.

Pro's

• Support multiple data sources
• Add new bands or even new data products without changing structure

Con's

• We'll have to strictly follow the naming convention <scene-id>_<band/image-name>.

Copied from original issue: phl-microsat-dpad/libra#1

[benjiao]

• another issue we'll probably encounter if we rely on the naming convention would be filtering by band/data products available

[benjiao]

From @rukku on November 25, 2015 11:8

@benjiao We'll have to consult with project 5 on the types of data products. diwata 1 also has three sensors:

1. SMI with LCTF
2. High Precision Telescope
3. Wide field camera

So that makes for at least three types of images. We still don't have enough information on how the bands would be named etc for the LCTF data, too. Diwata 1 scenes are also irregular so we'll have to devise a naming convention for it, too.

http://landsat.usgs.gov/naming_conventions_scene_identifiers.php

LXSPPPRRRYYYYDDDGSIVV
L = Landsat
X = Sensor
S = Satellite
PPP = WRS path
RRR = WRS row
YYYY = Year
DDD = Julian day of year
GSI = Ground station identifier
VV = Archive version number

Examples:
LC80390222013076EDC00 (Landsat 8 OLI and TIRS)
LO80390222013076EDC00 (Landsat 8 OLI only)
LT80390222013076EDC00 (Landsat 8 TIRS only)
LE70160392004262EDC02 (Landsat 7 ETM+)
LT40170361982320XXX08 (Landsat 4 TM)
LM10170391976031AAA01 (Landsat 1 MSS)

Landsat uses the path/row concept in its scene ids as its scenes are fixed. We'll have to use something else for Diwata. Though we can copy the other concepts.

[benjiao]

@rukku: Wow this is very helpful. I thought the characters after "L" was just a hash.

Re: naming convention for Diwata, I guess it would make sense to make our file structure more insensitive to file names.

We can probably do:

|   +-- landsat
|       +-- LC81180542015327LGN00
|           +-- thumbnail
|               +-- LC81180542015327LGN00_THUMB
|           +-- band1
|               +-- LC81180542015327LGN00_B1
|           +-- band2
|               +-- LC81180542015327LGN00_B2

Where thumbnail, band1, and band2 are keywords recognized by the webapp. This way, the file names they send us can be anything. Also to avoid dependencies.

[benjiao]

To add to my comment above, the link to be stored in our metadata storage will be /landsat/LC81180542015327LGN00. Links to the thumbnails and individual images can be easily derived by the landsat-api clone on the fly.

This way, we won't have to update our metadata storage everytime a dataproduct has been added/computed for a specific scene.

[benjiao]

From @lkpanganiban on November 26, 2015 6:14

So I have done some readings about image storage. It is recommended to store the images to a file system then store the metadata to our database. So naiisip ko yung structure ay may load balancer in front of our image file storage since maglock up yung connevtion kapag nagsimula magdownload si user.

[benjiao]

@lkpanganiban what LB setup do you have in mind? HAProxy? Squid?

That implies we setup multiple identical nodes for the file server kasi. It might affect our initial design for the update service AND double our space requirements. :p

Usually when we setup a download server, we just use Nginx. Maybe you have a better idea. :)

[benjiao]

From @lkpanganiban on November 26, 2015 8:13

okay na yung nginx for hosting our images. since we are going host landsat 8 images usually they are about 700-800mb per scene so kailangan talaga na madami tayo servers for just hosting our data. We can also do links via email. kapag nagrequest ka nang data isend yung download link via email para controlled yung number of downloads natin per day.

[benjiao]

From @lkpanganiban on November 26, 2015 8:35

We can also do that we will just direct our downloads to usgs servers. hehe. :)

[benjiao]

We can also do that we will just direct our downloads to usgs servers. hehe. :)

@lkpanganiban: Hehe. Actually pwede as failover. Wag nalang natin tanggalin yung links to usgs images from the meta. Though mas ok padin kung locally hosted so students and researchers can have faster downloads within campus.

Anyway, I think the best thing to do at this point is to finalize the directory structure so we can start working on the update/download script. Just to recap, we will have to consider the following:

• New data sources will be added in the future (Diwata, Diwata 2)
• New derivatives will be added for each scene
• Different data sources might have their own file naming conventions
• OODT might use the same directories as data source as well as output bins (@rukku, nudge nudge)

Pacheck nalang ng latest file structure na pinost ko above. :)