search

hashbangcode / vlad

Vlad - Vagrant LAMP Ansible Drupal
173 stars 53 forks source link

Make Vlad available via package manager? #253

Open zxaos opened 9 years ago

zxaos commented 9 years ago

While setting up a new project in the 2-repos-nested format, it occurred to me that this format might be significantly improved by handling the download of vlad itself through composer rather than a submodule.

I did a search through the issues and didn't run into anyone else who looked like they'd tried it. Certainly there'd be some work to get vlad to be happy running out of vendor/ (or using composer/installers) but I didn't see any major reasons this couldn't work.

Has anyone tried this?

philipnorton42 commented 9 years ago

I've not tried this myself, but it would only need a few paths changing. Mainly in the Vagrantfile I think. But seeing as it's not a PHP application I'm wondering if it would work inside packagist?

Are you just looking for a simple way to say 'use Vlad in this app' and run a script that would include Vlad into the correct place? I'm just wondering what the benefits are to using composer.

zxaos commented 9 years ago

I hadn't considered whether Packagist would take it, but it's a good point.

What I was hoping to accomplish was to simplify setup of new projects using Vlad, to avoid having to check in Vlad itself into my source repository, with the goal of still having a "clone this repo, run vagrant up, and start hacking on the project" simplicity in place for other team members who may not know vlad and just want to use it to develop a Drupal project.

Submodules in theory solve that in that 2-repos format, but in practice they're a poor solution because people forget to initialize them, or they don't understand submodules, or the inadvertently commit submodule commit version regressions. I suggested composer because we're already using it to manage the dependencies of the project-that-uses-vlad so it seemed like the simplest tool to suggest.

If something like bundler would make more sense, though, I don't know.

dixhuit commented 9 years ago

Another possible route here that I've been considering for some time is Yeoman (or more specifcally, creating a Yeoman "generator").

Yeoman is for "scaffolding" (buzzword!) projects (getting files & directory structures in place) and can include optional prompts when doing so. An example workflow would smell a bit like:

$ yo vlad

Yeoman would then spring to life and ask us a bunch of questions about how we'd like our project set up. The specific prompts are all up for grabs but some initial ideas include:

I'd like to see all prompts include defaults and would like to investigate ways to bypass prompts altogether.

Yeoman could move onto optionally kicking off VM provisioning (vagrant up) if appropriate.

Yeoman depends on Node and can be installed via NPM.

zxaos commented 9 years ago

@danbohea while that helps with initial project setup, how does it work for someone cloning the repo?

Would they just clone and do bower install? If they were to run yeoman again wouldn't any checked-in configuration get overridden by new files?

dixhuit commented 9 years ago

@zxaos That's true - I guess I didn't understand that that was what you were trying to do.

zxaos commented 9 years ago

@danbohea well, really, it's both. I end up usually being the one who sets up the projects so a yo vlad is useful to me but I also know enough to debug it and update it and make sure it all works. When I have a team member who doesn't care about tooling and just wants to hack on a Drupal site however...

Maybe this should have been two separate issues :-)

dixhuit commented 9 years ago

OK, to return to the original issue then (Yeoman tangent aside - agreed we should split that out into a separate issue though), I'm probably a bit biased towards Bower here:

Thoughts?

zxaos commented 9 years ago

Makes sense to me. Solves the specific problem, and uses a familiar tool. +1.

My only potential argument (and it's weak) is that starting with D8 composer is the default since core ships with a composer file. I think in practice, enough people will have both composer and bower installed and I agree that bower is a better fit from a language/tooling perspective.

philipnorton42 commented 9 years ago

Dan, since you are more familiar with Yeoman/Bower would you mind leading this task?

dixhuit commented 9 years ago

Never created a Bower package before but I'll certainly give it a shot :)

It sounds like Matt's pretty familiar too so perhaps he can help me test?

zxaos commented 9 years ago

@danbohea Can do. I have a couple of baby projects I would be happy to experimentally rebuild with bower rather than a git submodule for vlad. I'm exclusively working with the 2-repos-nested layout, but from my perspective that's really the one this is designed to replace anyway.

dixhuit commented 9 years ago

@zxaos Marvellous.

I've just been reading this and it seems that package definition and submission is pretty straight forward.

The main thing we're gonna need to think about is project structure. By default bower packages are installed in bower_components which in my mind would sit like this in a Bower/Vlad/Drupal project setup:

my_project/
├── bower_components
│   └── vlad
├── docroot
├── settings
└── vlad_aux

Point 1: Does this look right to you? I can't think how I'd improve that without unwanted compromise. If it looks right then we should probably consider changing the following variable defaults to:

aux_synced_folder: ../vlad_aux
host_synced_folder: ../docroot

Turns out that Phil and I were considering doing this anyway.

Point 2: The presence of a .bowerrc file that sets the value of directory to something unexpected could complicate this further. The name of the directory isn't so much of an issue here but the fact that it can be placed pretty much anywhere is gonna give us problems. Sure docroot & vlad_aux locations can be configured via settings but the possible locations for Vlad's settings files are still currently fixed within Vlad.

Point 3: As it stands, this setup would break if a custom role was present and configured to be used via Vlad settings. However, as soon as the custom-play branch gets merged this will no longer be an issue (custom roles will then be custom plays and can live anywhere, the path is set via Vlad settings).

So, presuming that point 1 and point 3 are all good that just leaves point 2 - location of settings files. For this to work I see a couple of options (please add any further ideas):

  1. We add a further possible relative settings location to Vlad (to cater for nesting Vlad within bower_components or similar) and then insist on a certain relative location for Bower packages to be installed to in order for Vlad to be expected to work (refer to directory tree diagram above). This feels a bit hacky but for all I know there are plenty of Bower packages in the wild that work in this way (or not considering that Bower is fairly frontend module/library focussed? I have no idea.).
  2. We come up with a better way to locate Vlad's settings. Ansible isn't the most flexible thing for this kind of thing - the mere fact that I had to add dummy_settings.yml is testament to that. The only thing I can think of here is to use Ruby. At the top of the Vagrantfile we could scan for Vlad settings files in a much smarter way and then copy them to a predetermined location that Ansible is expecting. This would give us way more flex but we'd still need to restrict the use of .bowerrc's directory option to a sensible range of possible locations to avoid having to potentially scan the entire host box's file system (just no).
zxaos commented 9 years ago
  1. I think you're missing a .. there. If the Vagrant file is at myproject/bower_components/vlad/Vagrantfile then to have settings and vlad_aux in the main directory you'd need those values to be ../../vlad_aux and ../../docroot. That could complicate things for non-bower setups since those haven't been supported paths up until now.
  2. Not only with .bowerrc, but with command line arguments to boot.
  3. This one's outside my expertise - I've just been reading the issue and trying to keep up.

Suggestions

I think that doing some additional ruby processing in the Vagrantfile to try to be a bit smarter about the settings location is a good way to go forward in general here though.

If you had the paths for the settings, would you even need to copy them to a predefined location for Ansible? Couldn't you just point it at the relative path calculated by ruby directly?

So I guess my preference would be: add some smarter settings file location code in the Vagrantfile. Check for some sane basic locations (for speed), then if we're in bower mode, start looking for a bower and/or a bower.json. I just don't want to go overboard with making this complicated to suit my own particular use case here (although you wouldn't know it :stuck_out_tongue: )

dixhuit commented 9 years ago

I think you're missing a .. there.

Right you are. It was late when I wrote that ;)

If you had the paths for the settings, would you even need to copy them to a predefined location for Ansible? Couldn't you just point it at the relative path calculated by ruby directly?

Currently these paths are set in Ansible YAML files. The only dynamic alternative to doing this is to pass variables in when running the Ansible play via the command line. These would need to be either inline in the command (which will require some formatting prep) or as a JSON file (which will require converting to JSON, oddly), Given the options there, copying and possibly renaming a couple YAML files is possibly the most straight forward route.

Smarter settings file scanning and processing via Ruby is something that would likely benefit the project even if we weren't trying to add Bower support:

I suggest we start with this task in isolation with a view to then taking it further with Bower in mind specifically (checking for Bower related files etc).

@philipnorton42 I don't know if you're following along here anymore. Any resistance to the proposed changes for handling settings files in Vlad?

dixhuit commented 9 years ago

Bower aside for a second, having just scanned NPM's code of conduct regarding acceptable package content it appears that Vlad wouldn't be unwelcome there (I've reached out for confirmation). This may be a more sensible alternative as NPM's node_modules directory is fixed (in name & location).

dixhuit commented 9 years ago

The above commit tweaks things so that Ruby creates a single merged settings file on the fly and Ansible always uses the same file in the same location. From here we can start thinking about a more sophisticated means of scanning for settings files if we need to.

zxaos commented 9 years ago

One upside to using something like npm would be the ability to wrap the calls to vagrant to sidestep having to be in the same directory as the Vagrantfile. So we could do npm run vlad up which could do cd node_modules/vlad && vagrant up or something. Given that bower is itself an npm package, you end up actually requiring fewer tool if it is packaged directly in this way.

A future consideration if we use npm would be what should happen if someone were to install vlad globally. As vlad moves toward support for custom roles and support for multiple installs per VM, that might provide a route to treat the entire vlad VM like a utility rather than a part of an individual project's structure - but I suspect functionality like automatic db dump import would be necessary to fully take advantage of that.

Edit: Also the code in 5b0ad98 looks much easier to read - I hadn't realized how many times the settings location stanza was included across different files.

dixhuit commented 9 years ago

One upside to using something like npm would be the ability to wrap the calls to vagrant to sidestep having to be in the same directory as the Vagrantfile. So we could do npm run vlad up which could do cd node_modules/vlad && vagrant up or something.

This is an excellent point, not to mention an impressive display of psychic ability (I was just having a very similar discussion around the need to cd to the Vagrantfile in order to run vagrant commands).

Given that bower is itself an npm package, you end up actually requiring fewer tool if it is packaged directly in this way.

Also a good point and yet another thumbs up for npm.

Regarding global installation, that is something that I've considered before (though not via npm specifically) and well worth further exploration. Right now though, let's park that point at "npm could provide a sensible route here, but let's not fully explore that right now" (anything less and my brain might melt).

OK, let's look at npm in more depth - maybe that'll give @npm_support enough time to check Vlad out and respond to my enquiry (or maybe that's just never gonna happen).

One thing that did occur to me is that unlike Bower, npm doesn't install packages side by side in a "flat" directory. That's not an issue if Vlad is only required directly as Vlad will likely end up living at:

my_project/
├── node_modules/
│   └── vlad/
├── docroot/
├── settings/
└── vlad_aux/

However if Vlad is listed as a dependency of another project that is required then Vlad could end up at potentially any level of crazy node_modules nesting:

my_project/
├── node_modules/
│   └── rabbit_hole/
│       └── can_of_worms/
│           └── the_downward_spiral/
│               └── vlad/
├── docroot/
├── settings/
└── vlad_aux/

I hadn't considered npm acting as a wrapper for Vagrant/Vlad before. Would that mitigate this issue do you think? I.e. Given that npm would have placed Vlad wherever within node_modules is npm therefore always aware of that location and able to make it easily available to us (I'm guessing/hoping yes)?

Even then, it's still currently up to the user to tell Vlad where docroot & vlad_aux are to be created/found. An un-amusing amount of nesting courtesy of npm could make this a nightmare - we may have to consider having Vlad "figure this out" which brings us back to scanning from the Vagrantfile albeit slightly easier than when originally considered with Bower:

  1. We'd start at the Vagrantfile and work our way up until we found settings/vlad_settings.yml - that would confirm the location of the project root.
  2. Users would provide directory paths relative to the project root rather than relative to the Vagrantfile.

My brain's hurting now. I this making any sense? :)

dixhuit commented 9 years ago

https://twitter.com/seldo/status/623875434141368320?cn=cmVwbHk%3D&refsrc=email

So that's public sign off from the CTO of npm which is good enough for me. Now we have options :)

zxaos commented 9 years ago

Ok, parking global discussion for now.

The npm docs state a bunch of configuration is provided to the scripts at runtime, but I'll need to do more research to see what exactly we'd get. I can't imagine there's insufficient information to correctly target a script since plenty of things are distributed this way, but that doesn't help with the specifics needed to implement.

One potential tool here would also be preinstall or postinstall npm scripts. If it's easier for vlad to just look at a "you installed through npm so here's where your settings live" blessed folder, then that'd be the place to make that happen I'd think.

Finally, I think if vlad gets installed as a dependency rather than directly, that npm doesn't register the script for it anyway. Consider something like karma which bundles http-proxy: the karma bin exists but the http-proxy bin is only present inside karma's directory. In other words, if some other package wants to wrap vlad, it becomes up to them to invoke it correctly.

dixhuit commented 9 years ago

In other words, if some other package wants to wrap vlad, it becomes up to them to invoke it correctly.

Agreed - that's fair enough. So with that in mind, can we assume that for our basic purposes Vlad will only ever live in node_modules/vlad/ if installed via npm? If so, that makes everything a bit easier and our next step is to simply add a further possible location for vlad_settings.yml and vlad_local_settings.yml so that we can get a simulated project structure working. Let me know what you think and I'll get this bit committed (I may create a new branch for this stuff).

zxaos commented 9 years ago

So just adding ../../settings ? Makes sense to me.

dixhuit commented 9 years ago

OK, this is now done (see the newly created "npm" branch).

I've just noticed that "vlad" isn't available "as is" on npm due to the existence of https://github.com/nickclaw/vlad. Damn you Nick Claw! ;)

So that's as good a next step as any: what do we call this? Apparently npm have a thing called scoped packages which looks to solve this. This in itself will require a username to be decided upon at the very least (it serves as a prefix much like GiHub: username/project) so for example we could still use "vlad" providing it's "scoped" with a username e.g: nosferatu/vlad

So, 2 options:

  1. Decide on a namespace that is available. This will inevitably be long and far from ideal (boo!). I did notice that "drupalvm" and "drupal-vm" are both available - is that naughty? JOKING! :)
  2. Plump for a scoped package in which case we need to decide on a common username. I'm guessing this would be "hashbangcode" to match the GitHub repo?

I'm thinking option 2 is the way to go here probably.

@philipnorton42 You're input is required here :)

dixhuit commented 9 years ago

Hahaaa! I just accidentally stumbled across #2 which I've never seen before. Amazing.

dixhuit commented 9 years ago

Just added an initial package.json & .npmignore:

https://github.com/hashbangcode/vlad/blob/npm/package.json

I've not published the package to npm yet thinking that you guys may want to give this the once over first.

zxaos commented 9 years ago

I'll be back on a project that I could justify re-building to use with npm on Monday. I'll just be requiring vlad directly rather than trying to do any sub-package testing or anything.

dixhuit commented 9 years ago

OK, cool but just to reiterate:

I've not published the package to npm yet thinking that you guys may want to give this the once over first.

zxaos commented 9 years ago

Right - I'll be testing by doing a npm install of the npm branch directly :-) That should help shake out any issues.

dixhuit commented 9 years ago

Right! Didn't think of that, good idea :)

zxaos commented 9 years ago

Ok, so it installs cleanly that way.

However, if I try to do something clever, like set up a command to VAGRANT_CWD=/path/to/vlad && vagrant up in npm, I get a pathing error. I can confirm it running the commands separately too:

~/d/vlad-npm-test ❯❯❯ export VAGRANT_CWD=/Users/matt/Developer/vlad-npm-test/node_modules/@hashbangcode/vlad
~/d/vlad-npm-test ❯❯❯ vagrant up

Checking for project settings and local overrides...
There was an error loading a Vagrantfile. The file being loaded
and the error message are shown below. This is usually caused by
a syntax error.

Path: /Users/matt/Developer/vlad-npm-test/node_modules/@hashbangcode/vlad/Vagrantfile
Line number: 75
Message: Errno::ENOENT: No such file or directory - vlad_guts/merged_user_settings.yml

This might be a separate issue with the new vlad_guts locations, but I'd say that running vlad with VAGRANT_CWD set would probably be an expected environment for this kind of setup - especially if we want to get to the point where something like npm run vlad -- up is a thing that works.

But it installs fine! :+1:

Edit: and the more standard cd node_modules/@hashbangcode/vlad && vagrant up runs still, but see comment below.

zxaos commented 9 years ago

It's also not reading a top-level settings file in that scenario - because instead of being in node_modules/vlad-package-name it's in node_modules/@hashbangcode/vlad.

I don't think it's going to be easy to rely on fixed paths, even this way, since the minute you install vlad from git rather than the npm repo (say, to hack on a specific branch ) you're breaking that ../../settings path.

dixhuit commented 9 years ago

Righto - good start, though :)

I hadn't catered for Vlad to be nested that deep so it's understandable that it's not finding settings from there. Looking at the paths you've mentioned I'm guessing that the package "scope" is what's causing the extra level of nesting. 2 options spring to mind:

  1. Don't bother with package scoping and come up with a unique name for this package on npm.
  2. Accept that package scoping is going to cause this and allow for settings to be located that far above Vlad itself.

Thoughts?

the minute you install vlad from git rather than the npm repo (say, to hack on a specific branch ) you're breaking that ../../settings path.

Not really, this can work with fixed paths just so long as there are a few of them for Vlad to try and we think of what they might be in advance. We're already doing this, the only path we're not currently catering for is ../../../settings/vlad_settings.yml (great grandparent). Sure, having a more intelligent system is still better but we can get away with this for now so long as we've thought about the possible permutations.

dixhuit commented 9 years ago

OK, not all the possible permutations of course - within the realms of expected, reasonable use cases.

zxaos commented 9 years ago

My preference would be a unique package name, but I don't have any real objections about the second other than a concern about how far that'll go up in a non-npm scenario.

(Consider, non-npm vlad in /User/username/project/vlad: you'd be looking for a settings.yml outside the home folder at that point).

dixhuit commented 9 years ago

My preference would be a unique package name

If we go this route, do you know if we are bound to that name for any commands we might choose to add later? E.g. If we do end up trying to wrap vagrant commands in a vlad command (short, easy to remember) will this even be possible if the package is actually called "vlad-drupal"? I really don't know what's possible here. Ideally I'd like to stay on track for adding a single primary command (vlad) and personally am happy to let that steer the package naming if necessary (scoped if necessary).

(Consider, non-npm vlad in /User/username/project/vlad: you'd be looking for a settings.yml outside the home folder at that point).

True, but now that we've handed over settings processing to Ruby, in theory we can write something that's smart enough to deal with that.

Running with a unique package name for a moment, I just did a quick check of what options are still available on npm based on ideas that came to mind easily. The results are below. Admittedly, not all of these are that great - feel free to suggest more if you think of any better ones:

Docs on naming npm packages here. I'm checking availability by just running:

npm view idea-for-package-name

Looking over the list above, I don't particularly like where this is heading. I think I still prefer the idea of sticking with a scoped package name ("@hashbangcode/vlad"). From what I'm reading, npm kind of screwed up here and probably should have started off with mandatory scoping (like GitHub) to avoid this problem in the first place. Over time and if npm is here to stay, I think more and more people will be forced to go the scoped route.

@philipnorton42 Do you have a preference here?

zxaos commented 9 years ago

@danbohea After thinking about it for a bit longer, I now agree with you about just using the namespace thing.

Regarding the settings file, then, the easy case here would be for vlad-in-npm-mode to not care where it's installed, and just search upwards until it finds a directory with node_modules/ and settings/ in it.

I still think it'd be good to make the VAGRANT_CWD environment variable work too. Should I break that into a separate issue?

dixhuit commented 9 years ago

I still think it'd be good to make the VAGRANT_CWD environment variable work too. Should I break that into a separate issue?

Yes please :)

dixhuit commented 9 years ago

Just pushed https://github.com/hashbangcode/vlad/commit/0ee0be017a3f6a05489b7876367ef70b12c40dfa to the npm branch which should now let you run Vlad from node_modules/@hashbangcode/vlad.

zxaos commented 9 years ago

0ee0be0 (a contender for one of the cutest commit hashes ever, if you try to pronounce it), works fine and locates its great-grandparent settings successfully.

I've also created #265 , which is important when vlad is installed this deeply.

zxaos commented 8 years ago

Just poking to double check status on this - is #265 a blocker for making this officially go?

dixhuit commented 8 years ago

Sorry, got massively distracted by Galaxy architecture stuff and "real" work :) I keep meaning to get back to this.

265 could be a blocker depending on the view we take ( @cue philipnorton42 ). It certainly isn't great to have Vlad nested as far down as it would be when installed via npm. Vlad's settings detection is pretty deep rooted and is a major part of much of the convenience and customisation that Vlad offers. Right now, the only solution to #265 that I see is to remove it completely :(

The only way forward I see for this issue right now is:

Sanity check: are we sure that Composer/Packagist isn't the way to go here?

philipnorton42 commented 8 years ago

I just had a look around the Packagist site for some packages that contain either Ansible roles or ansible packages. There are a few in there:

https://packagist.org/search/?q=ansible https://packagist.org/search/?q=vagrant

So ignore my previous statement about Composer/Packagist being PHP only. I was clearly wrong! Shall we look into the feasibility of including a composer package? Shouldn't be too hard to get started.

dixhuit commented 8 years ago

That's good news. Agreed we should check out Composer as an option before going any further with npm (which always felt a little odd IMHO).

dixhuit commented 8 years ago

I'm far less familiar with Composer than bower or npm. Probably better that someone else take the reins here. New branch?

philipnorton42 commented 8 years ago

The composer config is pretty simple (it's just a single file) but creating a new branch as a test is probably a good idea. I'll create a really simple composer.json file.

philipnorton42 commented 8 years ago

Ah, so I've created the composer.json file but I can't add the project to packagist until the composer.json file is in the master branch. I also tried to setup a 'repositories" tag to try and pull the code in manually, but it didn't work for some reason.

I have therefore merged it with the dev branch. I guess we'll just test it properly when the next version is released, but here is the salient commit: https://github.com/hashbangcode/vlad/commit/759599428a7697e0cde451b969f6b933e20e00c6

zxaos commented 8 years ago

Requiring it with composer (in the dev branch) works on my end. Testing with some actual settings momentarily.

zxaos commented 8 years ago

.... but it's not seeing a settings/vlad_settings.yml file beside the root composer.json (where you'd expect it to). Testing with 0ee0be0 applied to dev.

philipnorton42 commented 8 years ago

@zxaos Interesting... Could you paste the composer.json file you are using? At least the relevant bits?

zxaos commented 8 years ago 
{
    "name": "matt/vlad-test",
    "description": "test",
    "license": "MIT",
    "authors": [
        {
            "name": "Matt Bond"
        }
    ],
    "minimum-stability": "dev",
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/hashbangcode/vlad.git"
        }
    ],
    "require": {},
    "require-dev": {
        "hashbangcode/vlad": "dev-dev"
    }
}