differentsmoke
commented
6 years ago Well, currently each XML file stores a single day like this:
<item>
<date>2017/01/03</date>
<day>Tue</day>
<time>12:36 AM</time>
<pred_in_ft>2.84</pred_in_ft>
<pred_in_cm>87</pred_in_cm>
<highlow>H</highlow>
</item>
<item>
<date>2017/01/03</date>
<day>Tue</day>
<time>07:03 AM</time>
<pred_in_ft>0.04</pred_in_ft>
<pred_in_cm>1</pred_in_cm>
<highlow>L</highlow>
</item>
<item>
<date>2017/01/03</date>
<day>Tue</day>
<time>01:00 PM</time>
<pred_in_ft>3.15</pred_in_ft>
<pred_in_cm>96</pred_in_cm>
<highlow>H</highlow>
</item>
<item>
<date>2017/01/03</date>
<day>Tue</day>
<time>07:49 PM</time>
<pred_in_ft>0.05</pred_in_ft>
<pred_in_cm>2</pred_in_cm>
<highlow>L</highlow>
</item>just by pre-processing the files and converting them to something like this:
"2017/01/03": {
"12:36 AM": [2.84, "H"],
"07:03 AM": [0.04, "L"],
"01:00 PM": [3.15, "H"],
"07:49 PM": [0.05, "L"],
}should save us A LOT of space. (Between 4/5s and 5/6s of the space).
But we could take a step further and do away with the static files altogether and store the information on the DB. The idea of the rake task I think is good for this. We should probably take Bryce's original script as a starting point.
susanfiu
The Current 2017 tide data is ~138MB I am assuming that 2018 is going be about the same, adding it will bring the size to 1/4 GB I highly recommend we put this data elsewhere.
Possible Solutions
Separate them into their own repositories and include them as Sub Modules.
Github - Working with Submodules Heroku - Resolving Application Dependencies with Git Submodules
Create a rake task that downloads that data into a folder in public. This might not be compatible with heroku's filesystem, more research will have to be done on that end.
Rails - Custom Rake Tasks