Feature Request: Simple, Standard, Seamless, PHP Install Option

[astorm]

Tagging @piotrekkaminski, per his request, regarding the post quoted below.

The short version: Irrespective of other goals and plans form the product team, a user needs to be able to

• Navigate to https://www.magentocommerce.com/download
• Download the latest version of Magento 2
• Upload it to a quality shared host, VPS system, or copy it to a local LAMP stack setup (Stock OS X PHP, WAMP, MAMP, etc) that offers modern, PHP (5.5, 5.6, 7.x) and MySQL
• Install the software
• Have basic core functionality (product adding (with images), purchase flows, order review functionality, etc.) work without errors, even if the host isn't rated for production levels of traffic

Without this ability, Magento will struggle to attract a significant number of new developers to the platform, and will lose a significant number of "DIY merchants" who have working class technical capabilities.

Regarding the specific post for context

Alan,

Thanks for the feedback! We treat all constructive feedback as a gift, whether praise or criticism. It is one of the strengths of the Magento community, which we always welcome as it helps us make Magento better for all of us. We do our best to merge community feedback into our product roadmap and plans. As a Magento Product Manager, I’m responsible for developer experience so let me provide some thoughts and feedback on the topic.

First, I fully understand your point of view and see the issues the same as you, as it relates to the complexity of Magento. We are well aware of this.

We have essentially four main groups of technical users personas:

• Do-It-Yourself (DIY) merchants - all those Community Edition sites with users who hang out on forums, where owner installs the software.

• New developer - someone that is new to Magento and not an advanced developer. Someone who could hack some changes in a Wordpress install and now wants to do Magento.

• Experienced developer - someone that has good CS knowledge but no experience with Magento, e.g. someone coming from the world of Symphony.

• Experienced Magento Developer - someone that has experience on the Magento 1.x platform, who may be certified and who works for a system integrator or performing freelance Magento development as his primary source of income.

There are also front-end developers, but let’s leave that topic for another time as we have work to do specifically for front-end developers as well.

The last two groups, experienced developers, might be able to overcome most problems if motivated. However, based on the feedback we are getting, it is clear that the there is too much friction with installation, especially for the first two groups, which is not what we want the experience to be.

Since launch, Magento 2 has tried to be very agnostic about the tools and flows. (We actually received community feedback during Magento 2 development encouraging us to do so.) However I think it is time for us to change that and clearly show “this is the way we think it works best.” If someone has knowledge/desire to do differently, that route will of course be possible for them.

Alan Kent outlined our plans in his article http://alankent.me/2016/04/02/... but I wanted to expand on that a little bit. We are now doing and planning to do the following to address the needs of each group.

DIY Merchant

They want to have their site up and running easily to start selling. Most if not all customization would be delivered through extensions and themes.

We are working to ensure that they’re able to use popular web hosting providers to set up a site with one-click. This is working great on Digital Ocean and recently on GoDaddy Cloud. The same can’t be said on other platforms and we are trying to improve it with changes on our side and in coordination with the popular web hosting partners.

Developers have largely separate needs.

New Developer

They want to be able to quickly start a site and be able to do some changes--simple extension or customization development.

For this group, I would like to have similar experience to what you get currently with Wordpress VIP: https://github.com/Automattic/.... Three commands and you can have a local development environment up and running. For this, we are internally working to evaluate the best approaches (Docker? Vagrant? 1 or more images?). It is not production ready - but great for local development. We would like to be able to recommend full development workflow - from IDE to dev box with minimal installation and hassle. This workflow might not even require anything to install locally – e.g. use an online editor. We currently have sample Vagrant and Docker containers available but we need to collect more feedback before we recommend any of them and incorporate into official documentation.

Experienced Developer and/or Magento Developer

They want to do more serious development on extensions or complex code that might or need to end up in production. The target experience is a mix of mentioned before Wordpress QuickStart with Acquia Dev Desktop. Again, we want full workflow, this time from IDE to production deployment. At Imagine, we will be announcing some changes that we feel will be welcome by developers looking for a better workflow. We are also considering adding some additional support for that into PHP Storm to make it easier to manage and deploy to staging. Again, there is some internal work to evaluate the best approach and technology (single container for everything or separate web/php/DB etc). The workflow has to support also running tests prior to any deployment.

Hopefully this gives you some idea what we are planning to do on the product side. In addition, we will continue to increase efforts around documentation and education as we believe this can also help lower the learning curve for new developers. Please let us know if you have any questions and I also welcome any suggestions to incorporate the community’s voice into the plans. The easiest way to leave feedback is to use GitHub issues and tag me (@piotrekkaminski).

It's always nice to see and hear Magento's thinking on where the product's heading, if only as a reminder that there are many people across many teams working on the product.

It's always good to hear the internal team's perspective on where Magento is at and what future plans are -- if only to be reminded there are people working on this, and are people thinking about it. A lot of the above sounds great, and exactly the sort of things that should be done.

That said -- there's a few disconnects in the user groups/stories you've identified. I'll outline them below in individual comments.

[astorm]

First -- there's a significant number of DIY merchants with technical skills -- either their own, their employees, or their offshore networks -- who have deployed their own stores, and maintain them. When these merchants take the current Magento 2 out for a spin and can't get it running, the platform is going to take a credibility hit.

[astorm]

Second, the stated goal of

"Most if not all customization would be delivered through extensions and themes."

is admirable, and a good long term goal, but is not realistic in the short and medium term. The above scenario is rare with Magento 1. The extensions and themes that exist now for Magento 1 are great, but many still require small code changes, adjustments, and extensions to meet the business needs of store owners. If extension and theme vendors need to tighten up their code and feature set to run without those adjustments (while converting to Magento 2 at the same time), they need a user base and revenue stream first to fund that development.

[astorm]

Third -- there's a gap in your developer profiling. Between your Novice developer who can hack on Wordpress, and your Experienced developer with a CS background who can work with advanced OO concepts in frameworks like Symfony or Magento 2, there are a huge number of smart, hard working "full stack" PHP developers who do the yeoman's work of making systems run. "Intermediate developer" is the HR term. These women and men work independently, or for agencies not affiliated with the Magento partner program. These developers are busy, constantly evaluating new technologies, and if Magento 2 can't be easily installed they will look elsewhere and use/recommend other platforms.

[astorm]

Fourth -- support for tools like docker and vagrant are great and important for long term platform growth and health, but for full stack PHP developers these tools are nice add ons to the already well established, industry standard, decade old practice of

1. Download archive of source code
2. Copy to apache based PHP server,
3. Run through GUI installer to set basic configuration/check permissions/bootstrap database
4. Use software

[astorm]

Fifth -- getting hosting companies on board with 1-click installers is also great, and also something that will help platform growth. However, if this is the only way to get a stable installation it greatly limits the number of people who will give Magento 2 a spin. Offer someone a free giant bag of candy and they'll take it -- charge a quarter for that same bag and fewer will take you up on the offer. Even if these one-click installers are on inexpensive hosting, the fact its commercial hosting at all will limit who tries it.

[astorm]

Sixth -- regarding numbers four and five above -- I'm sure you've noticed the huge challenges involved in getting Magento 2 up and running with web host's 1-click systems, or event on vagrant and docker. I'll bet dollars to doughnuts that having a version of Magento that's an archive of source files that can be uploaded to a standard apache mod_php web server and easily installed would make everyone's job there a heck of a lot easier.

[astorm]

Seventh -- Wordpress VIP is an interesting system to model -- but Wordpress VIP can get away with having a non-traditional installer process because people are already bought-in on standard Wordpess, which they can still install easily by uploading an archive of source files to a standard apache/php/MySQL hosting platform.

[astorm]

Wrap Up -- All the new developer initiatives Magento 2 brings to the table are super exciting, but it seems like a bad idea to assume these new technologies are going to be replacements for older, tried and true methods of installing PHP based software.

The best part? Making your software easy to install in the traditional-PHP-application-sort-of-way makes adopting all these newer technologies an order of magnitude easier. It doesn't need to be an either/or situation. You can have a docker installation but still provide a simple, working archive to end users.

Put another way -- consider this thought experiment. If the team's opinion differs from mine -- that some version of "That's the old way, this is the new way" keep cropping up, then I'd challenge you to remove the downloadable archive from from https://magento.com/products/community-edition and replace it with a link to the GitHub repository. If these new technologies are really ready to shoulder the burden, then that downloadable archive should be obsolete.

If that downloadable archive is not* obsolete [Hint: it is not obsolete], then Magento needs to get engineering and product resources unified and working together to make that archive behave the way any other standard PHP application distribution would, and continuously improving the installer experience. I know a lot of that is tedious, detailed oriented grunt work, but getting those tedious details right is what makes something a great open source platform.

[piotrekkaminski]

Thanks @astorm for the feedback. This is great stuff! The intermediate developers is a less visible group so thanks for pointing that out.

[alankent]

Agree with your executive summary at the top. There are ways to do it today, but we need to push them into the developer documentation so the installation instructions for a default new user are simple. E.g. make the Vagrant box we publish official, and in the docs. Any feedback on that Vagrant box is welcome. (For example, we have a Bar Camp session talking about using the Vagrant box with GoDaddy at Imagine.) Things have been improving here, but I agree it needs to improve more. We are continuing to work on it.

• “First” – yes. The default guidance in documentation should be the most reliable way. I personally like Docker, but it feels Vagrant is more common for development. (My personal question here is Docker Toolbox is actually easier to install than Vagrant. I want to finalize the Vagrant vs Docker as the “standard recommended solution”. Right now I think it should be Vagrant as that is more common, and lots of our internal developers chose this route.)
• “Second” – yes, there is vision and there is short term reality. We need to take both into account.
• “Third” – agreed. We can include that in the personas (as Piotr already noted).
• “Fourth” – half agreed. It depends on what developer band you are after. If a larger scale solution partner is moving to M2, I would set up the expectation from day 1 and use Vagrant/Docker as it will help developer productivity. It is common such developers need to switch between projects on different versions of the stack and this is what Vagrant/Docker solves. This does not mean I think base installs should be hard!! They should not be. But I think there is a clear trend, with benefits to developers, using these newer technologies. They are easy to use for small to large developers alike. BUT! there are pain points still with the new technologies in terms of IDE integration and performance. I have tried lots of variations here trying to work out the best approach.
• “Fourth” – there is another aspect as well. I think the development experience should cover development to production. That means it is not sufficient to only talk about setting up a local development environment. I would go further and say it must take into account how to take the result to production. They are linked. Using say GoDaddy + Vagrant box you don’t start from the code base – you start from your production instance, and pull that code to a development environment, make changes, and push it back again. I think we should expand the experience to the full workflow.
• “Fifth” – agreed (but I note there are free offerings around as well). In my mind there are phases of developers – someone just wants to play, someone who wants to write a theme, someone who wants to write a module, someone who wants to build a simple site with extensions from the marketplace, etc. We do talk about the different steps on the journey for developers internally. We have been refining this description. It will never be perfect, but we need to make the various types of users get on board easily. They have slightly different needs. We are mapping this out (but have not talked about it much externally). But we need to map out the different tools and provide better guidance on what tool to use when.
• “Sixth” – I somewhat agree. Its not that hard to get a Vagrant box running. We have a CLI to do various commands. But I agree there are too many rough edges with the CLI commands we have. We need to think of the tools more in terms of end to end development flows, not in isolation.
• “Seventh” – I am not a Wordpress expert, so no comment from me.
• “Eighth” – hey, you did not mention documentation! Documentation and education. Multiple times I have found nice aspects to the M2 design - undocumented. We are improving here compared to M1, but more is needed.
• “Ninth” – we need to continue to improve security as well as developer experience. “Security is the opposite of convenience” is a common security mantra. We need to find a good balance between the two.

“Are we there yet?” There are known friction points that we are working through. They get set priorities like all the other work. I think there is going to be a big inflection with the launch of Marketplace, the availability of low cost hosting, as we standardize on the Vagrant box, etc. And yes, we made a few mistakes along the way (which we are fixing). Yes we need to continue to push this forwards – close is not good enough.

The good news is I think this is not so much a change of direction as bringing a number of strands of effort together in a more coordinated, holistic way. And then communicate it clearly.

[astorm]

@alankent, I have a tremendous amount of respect for what you've helped build with the engineering team, and can only guess and wince at the myriad challenges, pivots, and conflicting priorities you must deal with everyday.

But this

Its not that hard to get a Vagrant box running

is just wrong -- or wrong enough from certain important points of view. It also tips towards the worst sort of developer machismo.

It is reasonable to expect an employee, contractor, or someone with some level of existing financial commitment and dependence on a software project to

1. Wade their way through documentation for four different install methods and figure out they need a vagrant box
2. Install Virualbox or another VM provider
3. Install Vagrant
4. Spend a few days trying to figure out which pre-built vagrant VM
   • Most closely matches their deployment enviornment
   • Will run on their networking without IP conflicts
   • Will have adequate file sharing performance on their OS (to NFS or not NFS)
   • Is likely to be well supported in the future
5. Or, if they're a little more savvy, put together their own provisioning
6. Continue to perform the usual VM system administration that's required over the life of a project

But is is not reasonable to ask the same of open source contributors who aren't familiar with the project, and may not be familiar with the above technology stacks. They will not go through it, and while these folks aren't Magento Inc.'s primary market right now, they are a vital underpinning to Magento's continued success as an open source project.

Magento 1, Drupal, Wordprss, Zend, Concrete 5, Joomla, Baïkal CalDav ... the list goes on and on. Successful PHP open source projects have always offered an archive to download and install, and have spent time and resources improving and refining that install process. Even abstract programming frameworks like Symphony and Laravel that have gone composer first were only able to do this after they had grown an audience by appealing to more than professional software engineers.

And, again, this isn't an either/or situation. You can still provide a top of the line modern software development enviornment for your engineers, and, at the same time product a well tested archive install build artifact aimed at full stack developers, tech savvy store owners, and IT departments.

This well tested archive and install build artifact will make life tremendously easier for the teams getting one click installers up and running.

This well tested archive and install build artifact will also make life easier for the teams (I hope?) that are working on an officially supported Homestead like ready-to-go vagrant box.

I think that's all I have to say about this online for now. To folks inside Magento who are reading this agree, thank you for fighting the good fight :)

[alankent]

Hi Alan, point taken. And community feedback definitely does help us drive internal priorities - its always welcome. I have been experimenting with a simpler flow for development on GoDaddy nodes (blog coming soon). I have it working, but there are definitely gotchas that I want to iron out. I think there is a good solid dose of fixing some rough spots then simplify documentation accordingly.

[joshuaswarren]

I just want to mention that we had someone who plans to be at the Docathon at Imagine this weekend mention wanting to help develop a better guide for the exact process you outlined, @astorm. Specifically, this process:

Download archive of source code Copy to apache based PHP server, Run through GUI installer to set basic configuration/check permissions/bootstrap database Use software

They mentioned to me that they've been running through this exact series of steps on some basic shared hosts and documenting all the trials, tribulations and workarounds.

I think this might actually cross between a Docathon project and a Hackathon project, since there may be some shims and utilities that could be developed to make this easier. And that's perfect, because we're colocating those two events this year to encourage exactly this sort of cross-polination. So, if you all don't mind, I'm going to point those individuals to this issue here and also refer to it on the Hackathon project ideas list. If you're going to be at Imagine this weekend or even just available online to serve as a sounding board on some of these issues, it would be much appreciated.

[piotrekkaminski]

Sign me up for that! Who is that someone?

[astorm]

@joshuaswarren Happy to jump in an IRC or Slack channel if I can find the time (and such a thing exists). Send me the detail zero and ones in some form :)

[beejhuff]

@astorm Alan, could you email me at bjh438-git@yahoo.com - I need your email address to send you the Slack invite.

[davidalger]

@alankent / @piotrekkaminski There is one thing I would like to bring to your attention here while we're on the topic of easy access for developers / end-users alike. Something I haven't seen mentioned in the above outlines (unless I missed it) is the ease of actually getting a copy of the software to begin with. The easiest method of getting the software is cloning from GitHub, a practice which is discouraged unless one desires to contribute back to the core.

Two exercise might make this point a bit more obvious, assuming one can pretend they don't already know how to do the Magento side of it.

Install Symfony

1. Navigate to http://symfony.com and find the download page.
2. Go to the download page and follow the instructions to install Symfony.
3. Jot down how long this took and what you consider the level of effort required was.

My install procedure looked like this and took me all of 5 minutes to have the software on my computer and ready to dig into. It even presents me with exactly where to go for docs (as if it needed an added bonus). So this one is very simple!

Install Magento

1. Navigate to http://magento.com and find the download page.
2. Go to the download page (assuming you can find it) and follow the instruction to install Magento. Wait… no, can't do that yet!
3. Register for an account and find your way back to the download page.
4. Select a file-format and download the software.
5. Expand the software
6. Start hunting down the documentation pages and references necessary to actually install the software.

It's not much easier installing via composer, which also requires users go through a mess of steps to create accounts, generate authentication information, all before finally being able to run the composer create-project command that's supposed to make standing up a project simple.

What I would like to see included in the results of this discussion:

1. The https://repo.magento.com/ server allow anonymous access to the CE packages so the software may be installed / updated without extraneous hassle, allowing for a download / install process as simple as the one I noted above with Symfony.
2. Downloads from http://magento.com/download work without being logged in. It used to be this way, but apparently someone thought it a good idea to add a barrier making it more difficult to download the community versions of the software.

These two seemingly simple things will help reduce the barrier to entry considerably.

[piotrekkaminski]

@davidalger both of those points have pretty good reason behind them. 1) We assume one will later use Magento to download updates or extensions. Getting repo access first ensures the experience later is smoother. 2) It helps tremendously with security, it allows us to contact people who downloaded buggy or insecure versions. Any idea how to solve for those 2 requirements?

[joshuaswarren]

@piotrekkaminski, @davidalger -

I think that the first point you make, Piotr, points to a bit of a fractured/contradictory approach here. Or maybe just a missing persona in your planning? I feel like a lot of assumptions have been made about the number of people who will be updating and upgrading via the web interface versus via an established dev workflow.

Given that Magento is open source, and all of the CE modules on repo.magento.com are open source, I think it's only a matter of time before someone setups up a public Toran proxy or a similar mirror that allows anonymous access to all of the CE modules on repo.magento.com, similar to the old OpenMage (I think that's what it was called?) efforts that provided a Magento 1 mirror on GitHub. This is not the first conversation in which I've seen a number of devs really getting frustrated at the lack of anonymous access to repo.magento.com, and I've learned that the community will either get frustrated and give up, or they'll write a hack/workaround to get around the frustration. It could save everyone a lot of time and frustration if you could find a way to allow anonymous access to repo.magento.com - basically, give devs and users the choice of either following the recommended approach of authenticated access to repo.magento.com or taking the more DIY/"you're on your own" approach to using anonymous access to repo.magento.com. This could be solved by a prompt in the installation process that requests, but doesn't require, authentication credentials, and lets users know why it's better/easier if they provide the credentials, but doesn't require it.

On the 2nd point - that's a good point. But I think that could be very easily resolved by a mid-installation or post-installation prompt in Magento itself that pops up and asks them to provide an email address for a low-volume, security-only mailing list so that they can be notified of patches, etc.

[davidalger]

@piotrekkaminski, @joshuaswarren -

Seems this thread experienced the post-imagine effect. Totally forgot about the need to reply till I was too busy to do it ;)

First I want to note that I completely agree with what @joshuaswarren said in his last comment. He's spot on!

If the experience of using the Marketplace is the concern driving at gathering credentials as a barrier to installation, there are other approaches which solve the problem, and IMO would create a better experience for the merchant than they have today. The approach I'm thinking of here would involve presenting the user with clear instructions for linking their Magento installation to their Magento Marketplace account when they visit the setup wizard. This way, installation is allowed to remain simple, with few (if any) major barriers to standing up the software to play around with and use. But at the same time, there is a clear and consistent path forward for any merchant who desires to use the Marketplace.

Right now the experience for using the Marketplace post-install is easily fractured. For example:

1. What happens to a merchant when they install the software via a tarball which already has all the composer packages present? Are they forced to add credentials before using it?
2. What happens when a merchant installs the software using a 1-click install system from a hosting provider such as Digital Ocean or Go Daddy (I haven't tried this, so I honestly don't know)? Are they forced to add credentials?

I'm asking these questions mainly to point out examples of where the user experience is easily fractured depending on the method someone used to setup the software. The most common scenario where someone is currently forced to setup credentials in order to install is in the developer workflow of installing using composer create-project, yet that is also the one install method where the marketplace is the least likely to be used, and we just want to get the software up and running quickly.

To maintain a consistent user experience for marketplace use, the experience needs to be controlled post-install and within the admin area, not before when the method of installation could be so disparate. Get users up and running the software, then worry about getting their information!

The 2nd point you raise is a good one. What I would propose there is to include a prompt to sign-up for the security mailer post-login the first time a new administrative user logs into the system. This will have the added benefit of allowing all admins the opportunity to maintain awareness, not merely the individual who happens to be the chap setting up the Magento account necessary for installation. Another benefit here would be putting the prompt in front of everyone using the 1-click installs which don't require they go to Magento.com in the first place.

Currently the requirement for authenticating installs (CE installs via composer specifically) is highly annoying. Quite honestly, if I had the time, I'd very much be considering setting up a public Toran proxy to serve up packages, because these credentials are becoming quite the thorn in my side trying to build demo setups, quick install scripts, etc. In every case, I've got to have initial setup instructions for someone to create and account and go get credentials before they can get a demo going. An example of this would be here: https://github.com/davidalger/m2demo — I created this largely for those on our technical sales team to spin up a local copy of Magento 2 as a sandbox / playground for poking around in. It works great, provided each individual I setup on it goes out and creates an account to obtain credentials.

Anyhow, I hope I don't come off sounding like I'm complaining or being a bear, but I really do think that the choice to put authentication in the way of instilling the software is largely doing nothing but annoying developers (including myself) and failing to truly achieve what it sounds like the goal was in adding the requirement to begin with.

[mttjohnson]

We work in the eCommerce space as we are working with Magento. The sites built with Magento target an audience that want something. We try to build an experience that reduces barriers and increases the enjoyment for the audience to reach a specific goal of providing value to the audience, and potentially exchanging value with the audience.

If we can reduce the barriers to the audience that is interested in starting or experimenting with Magento or that are trying to keep moving forward with Magento we stand to grow the community in a healthier way. A lot of the things in the Magento 2 architecture have greatly improved and reduced some of the barriers developers have run into in the past, but the installation process seems to have introduced some new barriers that could easily turn people away before they ever get a taste of the experience and enjoyment that Magento has to offer them.

Asking a customer on an eCommerce site to register an account before they are allowed to browse the products and offerings is a significant barrier to attracting new customers on a site, and it would likely interfere to some degree with existing customers as well. If we can delay the need for acquiring some kind of registration in order to get Magento installed and usable I would speculate that we would reach a broader audience in those initial steps. During the process of standing up the software, installing it, and seeing the site function is part of that evaluation process as people get familiar with what Magento is, how it works, and what can be done with it. There is typically some amount of experimentation, browsing, trying things out, and test driving, before anyone is ready to commit to moving further, and it seems that until someone is ready to move forward, they aren't ready to move past the barrier of registering and connecting their installation with the marketplace.

There isn't much value in registering a magento account until someone is prepared or in a position where they benefit from what the account and association with their installation is able to provide them. The value of the account and associated installation as pointed out, seems to primarily land around security notices, and marketplace integration. That value makes sense once someone is running a live store, or is has committed some effort to building on Magento and want to get additional functionality than what the core of Magento has to offer by itself. Until they are to that point, the effort involved in overcoming the barrier provides little or no immediately value or incentive, and it would be easy for people to just abandon the experience right there because of the effort involved.

There is plenty of effort involved in a Magento installation outside of the account registration and association with an installation. If that effort is delayed and placed in closer proximity to the value benefit, I think it would provide a more balanced and enjoyable experience for attracting more of an audience towards experiencing Magento 2.

[astorm]

No update since May -- assuming this isn't a priority for the engineering and am closing out.

[SchumacherFM]

Statistics say that they need around 3-6 month to close an issue or a PR. So a couple of days to go for the six month ... ;-(