DISCUSSION: Storing Resources (in the repo or not?) #5

Open benloh opened 5 years ago

In GitLab by @daveseah on May 20, 2019, 10:46

MEME's Resource List can refer to several media types: PDFs, HTML files, and so on. These are displayed in an IFRAME in the browser. While the IFRAME can show non-local assets (e.g. stored at another server on the Internet), for purposes of archiving and distribution, particularly in a closed network, we want to include these assets in some way. However, storing large binary assets can be quite expensive, growing the size of the repo and making it ever-slower to deploy.

This is a discussion of how to manage ease-of-use with practical considerations.

Archiving Considerations

We may want to restore a particular version of the software based on commit hash or tag.

In practice, we may only need to do this for particular tagged releases, so perhaps we can design a fallback approach where resources are first checked in a local directory, and failing that they look for the remote directory. We had a comparable approach for PLAE's media caching where the code would fetch the remote asset then save it locally.

option 1: check in everything to GitLab and integrate GitLab LFS. Makes restoring easier, but at the cost of having to maintain a GitLFS server.
option 1a: move repo to GitHub, and then use GitHub's LFS service. Slightly easier, but space is limited to 2GB without paying more. Every revision to the files uses up disk space, as the old revisions are maintained.
option 2: Maintain an official offline server and save the working sets of files with a particular tagged release on GitHub, and providing a link to the file archive in the tagged commit description. Problem is that this means an extra server to host files, which itself may not be visible from a limited school network. The files would have to be unpacked locally. That said, it would be possible to write a utility to handle the unpacking for us.

In GitLab by @daveseah on May 22, 2019, 08:40

After thinking about this, I'm thinking option 3 is to just commit the files to the current repo on Gitlab. We will be creating a new repo on GitHub later; I think GitHub is a better solution for us overall than GitLab because we like the UI better there, and everyone is already over there.

So for now:

while authoring binary resources (PDF, images), refer to a URL outside of the repo so your changes won't pile up (Git saves everything, and that makes the repo get slow over time)
for broader testing, commit essential files to this repo so people can grab everything in one plae from a tagged release.

This problem of managing binary assets is still a problem with Git; I think we want to manage tagged assets that are frozen as part of a release. However, we want to make the source code installation as "one-click" as possible, which I think we could possibly do by making a release server. This is extra work, but we will likely need an external server anyway for Electron app updates.

Looking for feedback on thoughts of who/what/where this server would live and be called, and what features it should have to make it accessible to developers on the team. I'll prototype one in the next few weeks.

In GitLab by @daveseah on May 24, 2019, 13:52

I've confirmed with Ben offline that he'll check-in assets into the repo, as discussed above.

In GitLab by @daveseah on May 25, 2019, 07:11

CHANGE

Apparently even just a few assets are already 60MB, so this is not super viable.

Ben suggests copying the assets manually into the electron app. This could work, but I'd like to automate the process to make it less error prone.

A "gitignored" folder called resources/ is added to the repo structure. This will be essentially be a cache folder that is referred to directly by the app.
A public webserver is created containing the resources that will be cached.
An additional manifest file in the resource describes each item or is the template used by the application to know what to call things in the app. This creates an authoring opportunity.
A media downloader is added to the npm script, so the resources are downloaded and copied into the resources/ directory when the npm run package command is issued to bundle the app

In the meantime, we can go through the steps of creating it manually so we have something by the May deadline.

Merge Request !13 implements a first pass at this.

Basically put your assets in src/app-web/static/dlc. The contents of the static directory are copied as-is to the electron build built folder.

The dlc folder is ignored by git now.

So that is where we would put our non-repo big assets, which will eventually be handled by a node script.

You should be able to manually test it though by putting things into the dlc folder and trying to load them from the app.

Eventually the npm run dev command will check and make sure the local dlc folder is synched with a server somewhere.

The dlc approach has been merged with dev-jd/pilot so all resources for the pilot are now stored in the dlc folder, and the corresponding resource definitions in pmc-data.js have been updated.

To Test

All the resources in the Resource Library should open a resource in the src/app-web/static/dlc folder. The dlc folder is currently empty -- you'll have to download the assets from the server.

Check out ba20437d5810b427fd96219161d9f891e73ce31f.
Download http://meme.inquirium.info/v0.3.0-alpha/FishinaTank.html to src/app-web/static/dlc
Click on the (1) Fish in a Tank Simulation resource in the Resource Library.
The Resource View page should display a netlogo simulation.

The rest of the assets are in:

http://meme.inquirium.info/v0.3.0-alpha/...

AmmoniaInTanks.pdf
AmmoniaTesting.pdf
FishAllVariables.html
FishFighting.html
FishinaTank.html
FishSpawn_Sim_5_SEEDS_v7.html
FishStarving.html
FoodRot.html
RajForumPost.pdf
VetReport.pdf

We can put these in a manifest file once we decide on a data format (json, text, csv?).

In GitLab by @daveseah on May 27, 2019, 09:32

@benloh I think our manifest file might be similar to whatever file format you are using for defining resource templates. Where are those located?

@daveseah Resources are defined in pmc-data.js around line 352 starting with the definition of the a_resource variable.

So did you want a JSON array that you can just walk down and retrieve resources from, e.g.

{
  server: 'http://meme.inquirium.info/v0.3.0-alpha/',
  urls: [ 'AmmoniaInTanks.pdf',
          'AmmoniaTesting.pdf',
          ...
        }
}

Or perhaps more flexibly, just a simple array:

[
  'http://meme.inquirium.info/v0.3.0-alpha/AmmoniaInTanks.pdf',
  'http://meme.inquirium.info/v0.3.0-alpha/AmmoniaTesting.pdf',
  ...
]

(Though the problem with a simple array is it encourages scattering of reosurces across different servers/directories.)

As of 8/15/19, the resources have been moved to adm-data.js around line 159.

As of 8/30/19, all assets are now on https://daveseah.webfactional.com/meme/00INFO.TXT

theRAPTLab / meme

DISCUSSION: Storing Resources (in the repo or not?) #5

Archiving Considerations

To Test