plugin reading too much semantics into mv-storage

[karger]

I'm trying to do something I'm not supposed to with the plugin; it should work fine, but the way the plugin is "parsing" and interpreting the mv-storage attribute makes it break.

I'm using a nonstandard firebase collection to hold the data document for a mavo app. In particular, it's a nested collection, not at the root of the firebase. This should work fine; collections are collections, opaque to the plugin which just uses the firebase api for reading and writing them.

However, the plugin is trying to be clever about identifying a collection-name and filename by splitting the mv-storage url on "/". Since my storage url has extra '/' this breaks and the data goes to the wrong place.

I know this parsing is intended to provide some nice defaults for novices, so they can specify both collection-name and filename ins the url, or else specify only the filename and get the default collection name. But it would be great to find a way that doesn't block out some of the flexibility available in firebase.

If you want to keep the current approach unchanged, I think the following will add more generality without changing what is there:

• If mv-storage has only one /, then the thing after it represents a custom filename in the standard collection
• if there are exactly two / then it represents a custom collection and a custom filename
• if there are more then two, then after the last / is the filename and everything before it is a collection identifier (Firestore.collection() can accept a slash-separated path to a collection).

The current approach does seem a bit arbitrary though: why does giving only a single '/' in the database url mean the user is specifying an custom filename in the standard collection, as opposed to using the standard filename in a custom collection? I don't know if the github connector is similarly arbitrary in its interpretation of urls. I wonder if it might make sense to reduce the need for the user to understand specific url structure by adding some new mavo attributes like mv-collection and mv-filename and letting the plugin construct an appropriate url?

[DmitrySharabin]

Thank you very much for using the plugin and for giving me regular feedback. I appreciate that.

I like the idea of leaving the URL untouched and add a couple of new attributes that will let Mavo authors provide names for a collection and a file. Especially since we already have the mv-firebase-storage attribute to specify the name of the “folder” for assets. It seems to be rather easy to implement.

To sum up: 1) the mv-storage and other analogous attributes only accept databaseURL as a proper value 2) we can specify a collection (in other words, provide a valid identifier that accepts slashes) via the mv-firebase-collection attribute (default is mavo-apps) 3) we can also provide a filename via the mv-firebase-filename attribute (default is a Mavo app name).

Did I get everything right?

[karger]

You're accurately describing what I proposed, but it would be good to have Lea weigh in. As our number of plugins grows, it would be good for them to evolve common attributes and semantics. Ideally, if you suddenly decide to change to a different plugin you shouldn't have to change a whole bunch of arguments to the plugin. It seems especially unfortunate if you have to tell a different plugin the same "idea" in a different way.

I hadn't noticed the mv-firebase-storage attribute, and from the documentation I can't tell exactly how it works. What is its relationship to databaseurl, collectionname, and filename? Is it really the same as the mv-firebase-collection I suggested? There would have to be some overlap I assume---how do they interact if all are specified?

Another thing I'm unsure of is which of these attributes are "live" and cause the mavo to update when changed (as described for core attributes in the documentation index https://mavo.io/docs/ ). That would be nice to document somewhere for all plugins.

[DmitrySharabin]

After registering a new Firebase web-app (step 3 in the docs) we get all the services Firebase provides to us: authentication, database, storage, hosting, cloud functions, etc.

The plugin lets us use the first three. To be able to access them from our Mavo app, we must provide some parameters Firebase gave us during the setup process: apiKey, authDomain, databaseURL, projectId, storageBucket and some others. That's too much to make the plugin work. 🤯

Luckily, most of these parameters are interconnected so I can get all of them from the databaseURL value. That's why a Mavo author must specify only apiKey and databaseURL. Otherwise, nothing will work. I must add that I don't use the databaseURL value explicitly when manipulating an app's data and assets—I do it via API Firebase provide to me.

We store artifacts a Mavo app produces in two different places: data—in database, assets—in storage. So, mv-firebase-collection and mv-firebase-filename define where and at what name Mavo stores an app's data in JSON format in the database accordingly. And the mv-firebase-storage defines where Mavo stores an app's assets. In other words, it' a path to assets. But there is one thing to be aware of. The plugin will add another part to it (as the Backend class prescribes)—relative path to store uploads (e.g., for images, it will add images). Mavo authors can change it via the mv-upload-path attribute. E.g., if mv-firebase-storage="folder/subfolder" and mv-upload-path="pictures" all images will be stored in the folder/subfolder/pictures folder in the storage. If mv-firebase-storage isn't specified, it defaults to a Mavo app's name.

In that sense, the mv-firebase-storage is almost the same as the mv-firebase-collection you suggested.

To sum up, when we specify all three attributes (mv-firebase-collection, mv-firebase-filename, and mv-firebase-storage), we have maximum control over where and how Firebase organizes and stores our data.

I hope I answered your questions.

[karger]

Oh yes, I'd forgotten completely about the asset storage part. So mv-firebase-storage is really aimed at asset storage, and has nothing to do with data(base) storage. If that's true I'm not sure why you said it's almost the same as mv-firebase-collection, which is really particular to firebase's peculiar model of data storage.

Would collection be specified relative to the databaseurl?

What do we do about the fact that mv-source might also need to specify a collection and/or a filename? Is this an argument for keeping it all in the one storage url so we don't get an explosion of attributes?

I'm unclear what you mean about not using the databaseurl---isn't that something that needs to be provided as an argument to the firebase api calls? If there is info we aren't using, is there a reason we ask for it?

Backing up and getting very academic for a moment, I'd like to understand what we might consider common to most storage back-ends and what would seem to be exceptional to a specific back-end. We seem to want at a minimum to let people (i) specify a generic location and let mavo manage the collection of data files it stores for various apps at that location, defining both the collection and the filenames in it, and (ii) specify an exact location for the data file of a specific mavo. Then there are some variants where we want to let users specify the collection (as a name, or as a location?) and the filename, maybe one but not the other.

Right now this is all done through mv storage and through some assumptions/implicit rules about the url. This leads to some weird case where, e.g., certain URLs specified in the firebase mv-storage don't actually resolve meaningfully---because the collectionname is left out. And the ones that do resolve resolve weirdly, e.g. the mv-storage url redirects to my firebase console, but looking at the realtime database where there is no data instead of looking at the cloud firestore where the data is located. Is this all too much for a single attribute?

I wonder to what extent we want to specify things through URLs which the plugins will understand, and can manipulate if they need to and to what extent we want to express them semantically.

Looking at the existing storage back-ends, dropbox has the simplest possible api. You have to specify an explicit url for the location of the data file, because dropbox exposes no notion of human readable collection or file names. The gihub and firebase apis on the other hand lets you specify any arbitrary human readable collection name and also an arbitrary human readable file name. But they do so by letting you manipulate the mv-storage urls in back-end specific ways. We certainly want nice defaults for both of those when they are available. Should we consider making mv-collection-name and mv-file-name into things that would be respected by all backends that could, and that default to mavo-apps and the mavo app name respectively? For example, the local storage plugin currently doesn't seem to allow that flexibility---it would be natural to let the user specify the key (filename) under which an apps data was stored. This could simplify a user switching from one back end to another---fewer parameters to reformat.

On the other hand, the back ends are so different. Github offers all sort of combinations of repo, branch, filename that don't map naturally to a collection name and file name. So perhaps the attributes should be entirely back-end specific in which case they don't have to be consistent across back ends---the firebase plugin could have a collection-name and file-name while github does something completely different.

[DmitrySharabin]

I believe now I understand your confusion about the database URL and why it didn't work when you tried to use it to specify a path to a nested collection. It's all my fault and my bad design.

When I started to develop this plugin, I didn't know how to tell Mavo how it can distinguish the Firebase backend from the others. So I used the one Valter uses in his version of the Firebase plugin—via database URL provided by Firebase. And that was my first mistake.

Then, I decided to make my plugin configurable and let Mavo authors specify the collection name and filename by adding them to the database URL. And that was my awful second mistake because the resulting URL has nothing in common with the real URLs Firebase uses to store data and assets. I mispresented some key concepts. Now I see it. And I'm sorry about that.

Luckily, this plugin is a WIP. I think I must refactor it and introduce some breaking changes into it. Even though we have many questions to discuss and Mavo might have some changes in its core someday, I really want to make the plugin usable now. And then we can bring our new ideas to it during the next iteration.

So, here is what I am going to do:

1. We don't need databaseURL anymore—it's sufficient to know projectId so we can get and construct everything we need out of it. projectId is one of the parameters we get when registering our Firebase web-app.
2. To enable the plugin we still need apiKey, and as the mv-storage value we'll be using projectId. E.g., mv-storage="mavo-demos". I think that won't confuse Mavo authors because we already have a keyword as mv-storage value—local.
3. As you pointed out, besides mv-storage, we can have mv-source and mv-init, and we don't want to get an explosion of attributes to define collections names (including their paths) and filenames. So I'd like to let mavo authors use the following schema for defining data storage (source): projectId/path/to/collection/filename. Which defaults to projectId/mavo-apps/appName if a Mavo author provides only projectId. But what if a Mavo author needs to specify only one of them, either collection or filename? I think, for now, in that case, we could ask them to provide the full value: projectId/path/to/collection/filename.

I believe this approach would free Mavo authors from messing up with URLs, would make an app code more transparent, and still would save some flexibility for advanced users.

What do you think?

[DmitrySharabin]

If you like, you can give the new plugin version a try. I haven't released it yet but it's available for experiments via this URL: https://dmitrysharabin.github.io/mavo-firebase-firestore/src/mavo-firebase-firestore.js

[karger]

Dmitry, certainly no reason to take blame for, anything. With research projects it's always about doing it once to find out the right way to do it next time.

I like the idea of working off the bare projectId, because as you say that's something they definitely need to know so we might as well use it as a parameter. It's nice that this separates out other things---like collection name and filename---that might make sense with other plugins as well. And it seems appealing to save them having to understand the specific syntax of urls that don't matter to them.

But here's a wrinkle: what if I want to mix and match backends, e.g. have mv-storage use github but mv-source use firebase? I think that at present specifying a plugin says which connector to use for mv-storage, but how can we specify the connector to use for mv-source? For the github backend, the attribute always starts with github.com so we can know. But if we allow a bare project-id with firebase, we don't have the information about back-end. And what if the user wants to name their project "local" or "none"?

So unless we make significant changes to mavo, perhaps we should accept that part of the purpose of the url is to specify the back-end connector. Which would suggest that we should have a firebase-specific url in the mv-storage (or mv-source, or mv-init) attribute when we want to use firebase---with the schema you suggested, but leading with the firebase hostname. Having mv-plugins="firestore" does not tell mavo which back-end to use mv-storage; it only says to make the plugin available, while the question of which back-end to use is resolved through the url in the mv-storage/mv-source/mv-init attribute.

What would we then do about default collection and filename? Well, if the firebase api can detect whether a url points to a collection or to a document, we'd be able to decide whether the user has specified the filename or not, and use the default filename if not. But, if the url points to a collection, I'm not sure how to distinguish "this is the collection where you should store the document" or "this is the collection where you should create the (mavo-apps) collection within which you store the document.

Now if we did want to make significant changes to mavo, I could imagine writing special "protocol url" e.g. firebase://projectid/... and github://repo-name/... and so on. But that is for future.

[karger]

Also, I have found a way to make the existing plugin work with the app I'm building (using a top-level collection). So please give me a transition path before releasing the new plugin!

[DmitrySharabin]

So please give me a transition path before releasing the new plugin!

If I understood you correctly, the transition path is the following: as a value for mv-storage use projectId/top-level-collection/appName. Could you please give it a try with the dev version, and if everything goes smoothly, I'll release the new version?

[karger]

So before making this transition, what do you think of my concern that the projectId syntax doesn't itself convey information that the firebase backend should be used, so doesn't work for mv-source or mv-init?

[DmitrySharabin]

Actually, it works.

For now, the way I determine that the Firebase plugin is used is by checking that the value of either mv-storage, mv-source, or mv-init is not a URL, nor local, nor none, nor starts with the # sign. So I filter out all other possible backends. At least the ones we have now. I know, it might be a bit hacky. But It's the only solution I have found. Yet.

Unfortunately, a Mavo author can't use either local, or none as a projectId, but later we could change that. I really like the idea with a special "protocol URL".

[karger]

I guess it's fair to say that if you mv-plugin="firebase" then by default all of mv-storage, mv-source, and mv-init are interpreted as firebase if you don't specify a full url. But perhaps for full flexibility you should allow a full url even if not requiring it (this would also provide backward compatibility for all the many applications using the firebase plugin :)