Provide a better way to use the Meteor Tool without Mongo

[mitar]

Migrated from: meteor/meteor#4645

[dr-dimitru]

I have apps running without mongo, with next packages only:

meteor
underscore
ecmascript
blaze-html-templates
reactive-var
jquery
tracker
standard-minifier-js
standard-minifier-css

As you see there is no meteor-base, but meteor instead. //cc @ccorcos

[zimme]

It would be nice to be able to disable the start of mongodb when running Meteor. For example when you use Meteor as a build system/dev server.

[hwillson]

Or if you’re leveraging some totally random library to work with databases other than Mongo … 🙂

[stubailo]

Yep, I think this would be a huge improvement for people looking to use Meteor as a build system. Perhaps we can factor out the database stuff as a hook somehow, so that it can be implemented as a package?

[zimme]

So can we split this work up in at least 2 phases.

1. Allow Meteor to be run without starting MongoDB. MONGO_URL=false or something
2. Make Meteor packages that don't actually need MongoDB work without MongoDB

The way I'm using Meteor in a few projects is as a build system with no packages that uses mongodb.

static-html
standard-minifier-css
standard-minifier-js
es5-shim
ecmascript
shell-server
webapp
hot-code-push

In these projects I'm not using anything Meteor specific and just using Meteor instead of webpack.

[stubailo]

Right now passing MONGO_URL=false already works! It feels hacky though since it's an env var.

[zimme]

Re-reading your comment again @stubailo, I believe you weren't talking about Meteor packages that is relying on MongoDB without needing too. So my previous comment might have been unnecessary.

[zimme]

@stubailo, well... awesome! That's all I need 👍 So all my needs are solved by knowing that.

[stubailo]

Yeah there's not much we can do about packages over-declaring dependencies IMO.

[zimme]

Ok, so with Meteor already being able to start without starting MongoDB alongside it. This discussion, I guess, will be more about if we need to find a different way of doing it? Do we believe --no-mongodb or some other flag is better than using an environment variable?

[stubailo]

I think the ideal case would be that Mongo doesn't start by default if you don't have the "Mongo" package.

[hwillson]

Maybe we should step back a bit and clarify what this issue is about. The title says “Allow Meteor to run without a Mongo database”, but that is already achievable by removing the mongo package, and setting MONGO_URL=false (which I agree is quite hacky).

The intent of the original issue however (https://github.com/meteor/meteor/issues/4645), was to find a way to use Meteor to build web apps, without requiring Mongo, while still retaining lots of Meteor’s other benefits (like livedata, accounts, etc.). So not just using Meteor as a build tool, but being able to leverage certain parts of the Meteor ecosystem with different databases. Re-reading the original issue, I wonder how much of this is still relevant now that we have Apollo (and alternate ways of handling livedata for example). I guess the accounts part is still valid, but that should probably be tracked in a separate issue (if it isn’t already - I’ll check shortly).

So, what’s remaining with this feature request? Is it just coming up with a better way to disable Mongo, like https://github.com/meteor/meteor-feature-requests/issues/31#issuecomment-307178170?

[zimme]

@hwillson, I think that sounds about right.

[zimme]

So I'm thinking if we just check .meteor/packages for a mongo entry and if it's not there, set MONGO_URL to false. This way we don't have to rely on the code that parse the packages etc. which I believe is in the "app" which is started after mongodb.

[hwillson]

Yes, I think something like this could work. We'll have to keep in mind the upcoming https://github.com/meteor/meteor/pull/8769 changes (which will allow packages to be used via the --extra-packages command line parameter), but wiring up a check for this should be pretty straightforward.

[zimme]

I'll try and get a PR together later this week.

What I've found so far is that we should be able to use projectContext.ProjectConstraintsFiles getConstraint function to check if the project have the mongo package and it seems that the run command registration function in commands.js would be a good place for this.

[hwillson]

Thanks @zimme - I think we’re really close to moving forward with this, but I haven’t marked it as pull-requests-encouraged yet, as I haven’t had a chance to think about any gotchas we might be missing. Maybe hold off a day or two before working on this; I’ll aim to take a closer look shortly (and we can give others a chance to comment about this idea well).

[zimme]

No problem, wrote down what I found so far as I don't know if I'll have time to do this or not 😜

[zimme]

One thing that I can think of that might trip this up is the following. Can a package use MongoDB without using the mongo package?

[mitar]

What if we create a dummy package mongodb-dev-server which if it exists you start a server. And then mongo depends on it. Somebody can remove mongo, and add mongodb-dev-server if they just want to have it running.

[zimme]

I actually like that idea @mitar. I guess the PR would involve us finding core packages that uses MongoDB without depending on mongo, if there are any, and adding mongo-dev-server as a dependency on those packages, if we choose this route.

[abernix]

I like the idea of having mongo depend on mongodb-dev-server and having the server only start if that package is present. It at least makes the resource drain slightly more configurable (and sticky, per project).

Going a step further (probably too far), I think it'd be even better if mongodb-dev-server was not just a dummy-package but actually contained a lot of the (extensive) Mongo logic which currently lies in run-mongo.js, and maybe even the mongod executable. This would be a much larger undertaking so probably best to land the above recommendation first, then look further. It'd be nice to easily avoid the weight of mongod in the meteor-tool when it's not (ever!) going to be necessary!

As for implementation, I'm not sure that using projectContext's ProjectConstraintsFiles#getConstraint would work since that should reference packages which are directly listed in .meteor/packages. I believe the correct way to check would be to call projectContext.packageMapFile.getInfo('<packageName>'), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

[zimme]

Going a step further (probably too far), I think it'd be even better if mongodb-dev-server was not just a dummy-package but actually contained a lot of the (extensive) Mongo logic which currently lies in run-mongo.js, and maybe even the mongod executable. This would be a much larger undertaking so probably best to land the above recommendation first, then look further. It'd be nice to easily avoid the weight of mongod in the meteor-tool when it's not (ever!) going to be necessary!

Sounds good like a good direction forward.

Edit: I'm not sure this is even possible, MongoDB is started by the Meteor tool before the processing of packages are even started. Maybe it could be done with an npm package?

As for implementation, I'm not sure that using projectContext's ProjectConstraintsFiles#getConstraint would work since that should reference packages which are directly listed in .meteor/packages. I believe the correct way to check would be to call projectContext.packageMapFile.getInfo(''), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

That sounds like a better approach :+1: I'm keeping an eye on meteor/meteor#8769 too, to see how that plays out. I'm hoping it will be implmented in a way that makes projectContext.packageMapFile.getInfo('<packageName>') include the extra packages. So we don't have to do multiple checks.

[zimme]

I believe the correct way to check would be to call projectContext.packageMapFile.getInfo(''), which would properly check if it was included, as a dependency as well (for example, of a dependency of mongo, as suggested above).

It was packageMap that had getInfo, projectContext.packageMap.getInfo('mongo-dev-server');. This could work, but we would have to run projectContext.resolveConstraints() to have packageMap populated, I don't know if this is an expensive operation?

[zimme]

By using resolveConstraints() and packageMap.getInfo() we don't really have to wait for meteor/meteor#8769, as that will get the extra packages through eachConstraints once that PR is merged.

Also if we decide to check for the existence of a package and based on that start MongoDB or not, the developer needs to re-run meteor for it to take effect. This i have no problem with.

[zimme]

Looking into this a bit more, it would be better to use find/indexOf on projectContext.packageMapFile.getCachedVersions().keys() to see if mongo-dev-server is in the project. resolveConstraints() would add to the startup time which I don't think is good.~

There doesn't seem to be a better way than resolveConstraints() and packageMapFile.getInfo(), and the overhead of resolveConstraints() wasn't as bad as I thought because if used on it's own it just do the resolve part and doesn't try and download missing packages and the other stuff.

[zimme]

This is what I have so far while playing around with this. https://github.com/zimme/meteor/tree/zimme/mongo-dev-server

Should start mongodb if mongo-dev-server is present in the project. Shouldn't start mongodb if it's not present in the project. If the package is added while the app is running, you should get a message about restarting meteor to start the mongodb server. If the package is removed while the app is running, the mongodb server will still run and you still have the correct MONGO_URL in the environment. You have to restart meteor for the change to take effect.

[zimme]

Setting MONGO_URL should take precedence over mongo-dev-server i.e. it should not start a local mongodb server just as before.

[zimme]

I've opened meteor/meteor#8853 with a proposal that seems to work, based on the discussions in here.

I would love to have ddp decoupled from mongo so one could use dynamic-imports, which depends on ddp, without using mongo.

[ilan-schemoul]

Hello, I want to disable mongo so I did as you said I've set MONGO_URL to false. The server didn't start MongoDB which is a good thing yet it says it needs it :

> 32-PHARMACIE-METEOR@ start C:\www\eds-www\32-PHARMACIE_METEOR
> meteor run --port 2000 --settings settings-development.json

[[[[[ C:\www\eds-www\32-PHARMACIE_METEOR ]]]]]

=> Started proxy.
W20170704-00:00:00.070(2)? (STDERR) C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\meteor-tool\1.5.0\mt-os.windows.x86_32\dev_bundle\server-lib\node_modules\fibers\future.js:280
W20170704-00:00:00.125(2)? (STDERR)                                             throw(ex);
W20170704-00:00:00.126(2)? (STDERR)                                             ^
W20170704-00:00:00.129(2)? (STDERR)
W20170704-00:00:00.130(2)? (STDERR) Error: invalid schema, expected mongodb
W20170704-00:00:00.132(2)? (STDERR)     at module.exports (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\url_parser.js:18:11)
W20170704-00:00:00.132(2)? (STDERR)     at connect (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\mongo_client.js:401:16)
W20170704-00:00:00.133(2)? (STDERR)     at Function.MongoClient.connect (C:\Users\Ilan Schemoul\AppData\Local\.meteor\packages\npm-mongo\2.2.24\npm\node_modules\mongodb\lib\mongo_client.js:225:3)
W20170704-00:00:00.134(2)? (STDERR)     at new MongoConnection (packages/mongo/mongo_driver.js:169:11)
W20170704-00:00:00.134(2)? (STDERR)     at new MongoInternals.RemoteCollectionDriver (packages/mongo/remote_collection_driver.js:4:16)
W20170704-00:00:00.135(2)? (STDERR)     at Object.<anonymous> (packages/mongo/remote_collection_driver.js:38:10) W20170704-00:00:00.136(2)? (STDERR)     at Object.defaultRemoteCollectionDriver (packages\underscore.js:784:19)
W20170704-00:00:00.136(2)? (STDERR)     at new Mongo.Collection (packages/mongo/collection.js:99:40)
W20170704-00:00:00.137(2)? (STDERR)     at AccountsServer.AccountsCommon (packages/accounts-base/accounts_common.js:23:18)
W20170704-00:00:00.137(2)? (STDERR)     at new AccountsServer (packages/accounts-base/accounts_server.js:18:5)
=> Exited with code: 1

I've removed meteor remove meteor-base (previous versions of the technique was to remove meteor-platform but it doesn't seem to exist anymore...). And adding back meteor add meteor webapp hot-code-push (I don't use mongo but apollo so I guess I do not need the DDP package anymore). Here's my packages :

apollo                 0.9.1   � Add Apollo to your Meteor app
dynamic-import         0.1.1  Runtime support for Meteor 1.5 dynamic import(...) syntax
ecmascript             0.8.1  Compiler plugin that supports ES2015+ in all .js files
es5-shim               4.6.15  Shims and polyfills to improve ECMAScript 5 support
hot-code-push          1.0.4  Update the client in place when new code is available.
meteor                 1.6.1  Core Meteor environment
mobile-experience      1.0.4  Packages for a great mobile user experience
shell-server           0.2.3  Server-side component of the `meteor shell` command.
standard-minifier-css  1.3.4  Standard css minifier used with Meteor apps by default.
standard-minifier-js   2.1.0  Standard javascript minifiers used with Meteor apps by default.
static-html            1.2.2  Define static page content in .html files
tracker                1.1.3  Dependency tracker to allow reactive callbacks
webapp                 1.3.16  Serves a Meteor app over HTTP

Versions file :

accounts-base@1.3.0
allow-deny@1.0.5
apollo@0.9.1
autoupdate@1.3.12
babel-compiler@6.19.3
babel-runtime@1.0.1
base64@1.0.10
binary-heap@1.0.10
blaze@2.3.2
blaze-tools@1.0.10
boilerplate-generator@1.1.1
caching-compiler@1.1.9
caching-html-compiler@1.1.2
callback-hook@1.0.10
check@1.2.5
ddp@1.2.5
ddp-client@1.3.4
ddp-common@1.2.8
ddp-rate-limiter@1.0.7
ddp-server@1.3.14
deps@1.0.12
diff-sequence@1.0.7
dynamic-import@0.1.1
ecmascript@0.8.1
ecmascript-runtime@0.4.1
ecmascript-runtime-client@0.4.2
ecmascript-runtime-server@0.4.1
ejson@1.0.13
es5-shim@4.6.15
fastclick@1.0.13
geojson-utils@1.0.10
hot-code-push@1.0.4
html-tools@1.0.11
htmljs@1.0.11
http@1.2.12
id-map@1.0.9
jquery@1.11.10
launch-screen@1.1.1
localstorage@1.1.0
logging@1.1.17
meteor@1.6.1
minifier-css@1.2.16
minifier-js@2.1.0
minimongo@1.2.1
mobile-experience@1.0.4
mobile-status-bar@1.0.14
modules@0.9.2
modules-runtime@0.8.0
mongo@1.1.18
mongo-id@1.0.6
npm-mongo@2.2.24
observe-sequence@1.0.16
ordered-dict@1.0.9
promise@0.8.9
random@1.0.10
rate-limit@1.0.8
reactive-var@1.0.11
reload@1.1.11
retry@1.0.9
routepolicy@1.0.12
service-configuration@1.0.11
shell-server@0.2.3
spacebars@1.0.15
spacebars-compiler@1.1.2
standard-minifier-css@1.3.4
standard-minifier-js@2.1.0
static-html@1.2.2
templating-tools@1.1.2
tracker@1.1.3
ui@1.0.13
underscore@1.0.10
url@1.1.0
webapp@1.3.16
webapp-hashing@1.0.9

[abernix]

@NitroBAY apollo depends on accounts-base and accounts-base depends on mongo. With those dependencies, you'll need a MongoDB running for the Meteor.users collection.

[ilan-schemoul]

I've managed to remove apollo by duplicating only the necessary code from the package but meteor remove mongo gives 'mongo is not a direct dependency in this project.'

standard-minifier-css@1.3.4   # CSS minifier run for production mode
standard-minifier-js@2.1.1    # JS minifier run for production mode
es5-shim@4.6.15                # ECMAScript 5 compatibility for older browsers.
ecmascript@0.8.1              # Enable ECMAScript2015+ syntax in app code
shell-server@0.2.4            # Server-side component of the `meteor shell` command

static-html
meteor@1.7.0
webapp@1.3.17
hot-code-push@1.0.4

[ilan-schemoul]

On atmosphere I haven't found any of these packages to have a dependency to mongo yet it mongo is still listed in the versions file

[zimme]

@NitroBAY hot-code-push depends on autoupdate which depend om ddp and mongo

[ilan-schemoul]

Thanks.