Coverage Question Part 2

[blongstreth]

Thanks for the quick reply! Long term, I hope Meteor will be 100% npm so this issue goes away. For now, I need something that is easy, robust and allows for developers to build locally on their dev. boxes (which can be mac, linux or windows). With that in mind, maybe stratosphere is the right way to go?

If you think stratosphere could be augmented without a ton of work to cache the upstream stuff, would you please be able to provide some code snippets or an idea of all the touch points before I go off and review the code? Alternatively, I assume one could fetch the public package and locally publish to stratosphere. However, the downside is that version updates would be manual.

Finally, note that I am looking into caching the meteor installers as well since these should be backed up as well. This too will require writing a server if I want automatic upstream fetching :-) Possibly could add this to stratosphere.

Thanks in advance.

[blongstreth]

I think I have managed to get a good start on caching and serving the upstream files. I have more to learn about the server API and your code before I can call it a success (i.e refresh the cached files more intelligently, understand if the sync. process can be optimized/narrowed to only requested packages, serve the installer bootstrap files, etc.).

With that said, I want to thank you for cobbling up this project! I wish the MDG would have published something to help us folks out who need proper SCM. As I investigated how to do this, no other solution was better or easier on the end user than stratosphere, especially when you want to cache for developer and build machine boxes across platforms. In fact, until Meteor is 100% out of packages.meteor.com, this project will be useful assuming they don't change or break the API ;-) If I am able to get everything working correctly, I would be happy to send the code to you when I can. Maybe in a form of an archive or pull request if I get the time.

Finally, here are some things to note: Getting the code to run under Meteor 1.4+ was a no go. Had to use Meteor 1.3. The npm-bcyrpt package is not happy recompiling under osx and linux. I simply didn't have time to understand why. Down the road, maybe it would be a good idea to post an example project for those who just want to build, configure, deploy and go.

Thanks again,

Bradley

[sebakerckhof]

Hi, Just wanted to quickly say I'm currently working towards some deadlines and I'm very time constrained.

Success with the project. I hope I can be of more help soon.

[blongstreth]

No worries! I am just happy I am figuring things out!

[blongstreth]

One thing I need to look into which I am not familiar with is if we sync with packages.meteor.com and a package author decides to remove the package (marked unmigrated), we probably don't want it to go away in stratosphere cache. Seems like you removing packages is not possible and hopefully for good reason. Do you know off hand if this is true?

[sebakerckhof]

Well, the problem is that the Meteor package server doesn't really support deletion. If this happens (usually manually by a Meteor admin),then the lastDeletion value of the syncToken will be outdated. In this case Meteor will answer with a response that asks to reset the data. This basically means throwing away the entire database and re-syncing from scratch. See: https://github.com/sebakerckhof/stratosphere/blob/5ba4bdbcc5fc7fbd03c8520dd06ec04ef09f460f/packages/stratosphere_synchronizer/server/synchronizer.js#L100

You can easily change that behavior by always passing a lastDeletion value that's somewhere in the future.

[blongstreth]

Great news! I got it working with lazy file caching. Now I am on to fixing the UI. I am trying to get it to show public packages in addition to the private ones. Also, since the view is wonky in that it is missing some icon references and double rendering the navbar. The last item will be looking at caching the installers as well (maybe). I am also building a gradle plugin so I won't need Meteor command line or seperate installer (all done via gradle). Anyway, just an FYI on the current changes:

1. I refactored a few packages (i.e. upload packages into backend packages which have 2 servers: upload and upstream servers).
2. For caching I am doing something like this: When a new database element comes in from the upstream, I transform the url (rewrite to the local cache host) and check to see if the file hash code has changed for an existing entry in the local database. If something is new or changed, I set a flag in the element to indicate the file should be invalid and downloaded again (assuming it is present on the file system and accessed again). Also, I add a query parameter to the new file url which has some base64 encoded JSON metadata that is used to help look up the entry during file download request (i.e. has collection name, id and information about the file entry since 'build', 'versions', 'readme' entries are a bit different). If all goes well, it will either download or use the cache....Pretty simple actually. Only concern I have is what are the odds of the file getting out of sync on with the database entry. I rather not have to perform any hashing.

I wish the synchronization to the upstream database was more flexible in that only requested packages by clients are synchronized rather than the entire database. I am not sure if this is possible even if I was more apt with the Meteor server API.

Finally, Thanks again for your prompt replies! I have the code in a local cloned gitlab repo and when done, I will try and do a pull request unless you wouldn't mind settling for zipped source :-)

[sebakerckhof]

Cool. I actually started a big refactoring once to an angular 2 front-end and meteor 1.4 (es6 modules) backend, but I never got around to finishing it.

A more granular sync is not really possible I think, just because of the way packages are resolved locally.

I don't think you should care about cache entries being changed, since packages can't be changed. Only added or (very rarely) be removed.

[blongstreth]

Do you happen to have that angular 2 code? If so, would you consider sharing it? I updated everything to the latest Meteor 1.4+ but the UI is not rendering properly (nothing obvious). I think for now, the UI can be pretty minimal. Maybe a total package count, search, listing/filtered table and possibly an upstream refresh button. I am not sure if I need a UI initially to get things going.

[blongstreth]

Hello. I think I am finding that the synchronizer job may need to be converted into a single job which the results are shared by all connected clients. When database is empty, I find that if I start 2 or more clients, they all seem to keep reloading the same data. I assume you are aware of this, thus is why pre-loading the database is recommended so that so data duplication is minimized. True?

[sebakerckhof]

Do you mean synchronizing from upstream to stratosphere. Or from stratosphere to the client? In the first case, this could definitely be optimized. From stratosphere to the client, there I wouldn't really know how to optimize this, since each client would have a different sync token.

[blongstreth]

FYI: I have not worked on the code lately, but would like to report that I managed to successfully get my custom version of stratosphere which has full package synchronization plus platform installers to work really well. The only thing that is not working is the UI which I really don't use since I am not doing local packages (yet). Thanks for the help.