search

TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.65k stars 1.95k forks source link

Directory structure re-organization #448

Closed bhauer closed 10 years ago

bhauer commented 11 years ago

With so many framework implementations (it's awe-inspiring, really), the root directory is a little unruly.

Assuming it isn't especially difficult to do so, I'd like to re-organize the root directory structure slightly:

Leaving just README.md and .gitignore files at the root.

Thoughts?

kppullin commented 11 years ago

I like it. I'll further throw out the idea of grouping items in the /frameworks directory by platform/language.

fernandoacorreia commented 11 years ago

I agree. Regarding /toolset, I was going to suggest to create a directory for installation / execution scripts. But I think we should take it a step further and subdivide the /toolset directory by category.

The Azure deployment scripts I'm working on will require a directory of their own. I can envision a similar deployment script for AWS which would create the instances and automatically setup the environment. Those two environments could share some code (what is executed after the instances are created). This common code could be used to automate deployment for other environments (e.g. Rackspace, on-premises Microsoft System Center, etc.)

There is also setup for Linux server and client, Windows server and SQL server, each with their own bootstrap and auxiliary files. It's easy to see the /toolset directory getting out of hand soon.

One attempt to organize it:

/toolset: README.md for the toolset, run-tests.py /toolset/benchmark: benchmarker.py, framework_test.py, latest.json /toolset/deployment/aws: AWS deployment scripts /toolset/deployment/azure: Windows Azure deployment scripts /toolset/deployment/common: shared deployment scripts /toolset/setup/common: installer.py, setup_util.py, unbuffered.py /toolset/setup/windows: installer-bootstrap.ps1, installer.ps1 /toolset/setup/sqlserver: setup-sqlserver-bootstrap.ps1, setup-sqlserver.ps1

bhauer commented 11 years ago

@kppullin I like that idea of grouping by platform. It would further reduce the clutter in any single directory and help people find what they are looking for.

@fernandoacorreia That plan of further organization within the toolset directory sounds great too. If you have time and interest in making the change, I'd be happy to see it. If not, we'll get to it over here when we can find some time.

Thanks!

fernandoacorreia commented 11 years ago

I've made an initial pass at restructuring the toolset directory. You can see the commit in my fork and the forked branch.

I'm currently testing the changes and they may need a few adjustments in scripts due to changed directory names.

You'll notice that the "deployment" directory seems pointless now, but it will soon receive automated deployment scripts I'm working on.

bhauer commented 11 years ago

Looks like great progress. Thanks again for your continued contributions, @fernandoacorreia!

bhauer commented 11 years ago

Updated issue description at top to include a proposed layout for the Results directory.

(Aside: I don't know how to do more than 2 levels of bullet indentation in Markdown!)

serroba commented 10 years ago

I like the idea, but I think there should be as well a directory for the implementation using the nake language without framework

For instance, the PHP fremework test are using as base the implementation in vanilla PHP. I think in general is a good idea to have the vanilla implementation and a place to hold the frameworks, which should as well be aggregated on base of the platform/language

methane commented 10 years ago

@serroba I think frameworks/php/raw or frameworks/php/vanilla is enough.

serroba commented 10 years ago

@methane that's as well a good idea.

Actually the name raw is used across the project when using the DB against ORM or Raw.

I don't mind the name, but I mind that should be easy to read/understand for anyone. What I would like is that someone that is landing in the project, can start to write code quick and not to have to spend long time just trying to understand the tree and how things are organized.

bhauer commented 10 years ago

About "raw," @serroba makes a good point: I'd like to avoid using that term for anything other than the means of database connectivity to reduce confusion. To-date, we have used the word "plain" to refer to a platform without a framework. E.g., plain PHP, plain Servlet, plain Netty. But I would be okay with "vanilla" as @methane suggested as well.

However, I would prefer to organize the /frameworks directory by language first, and framework/platform second. Platform is a looser concept than language, and the precise definition of platform varies by language.

So for example, I would like something like the following:

/frameworks/php/plain   (or /frameworks/php/vanilla)
/frameworks/php/laravel
/frameworks/java/servlet
/frameworks/java/netty
/frameworks/java/spark
/frameworks/java/wicket

I think this would be awesome. Thoughts?

serroba commented 10 years ago

Although I like the most the term vanilla, I still remember a couple of years ago that I didn't understand what they were talking with the vanilla 'preposition'. As I no native English speaker I remember that I needed the explanation why vanilla.

Plain on the other hand is self explanatory. So thinking in the non native English speakers I think I would vote for plain instead of vanilla. On Jul 4, 2014 1:30 AM, "Brian Hauer" notifications@github.com wrote:

About "raw," @serroba https://github.com/serroba makes a good point: I'd like to avoid using that term for anything other than the means of database connectivity to reduce confusion. To-date, we have used the word "plain" to refer to a platform without a framework. E.g., plain PHP, plain Servlet, plain Netty. But I would be okay with "vanilla" as @methane https://github.com/methane suggested as well.

However, I would prefer to organize the /frameworks directory by language first, and framework/platform second. Platform is a looser concept than language, and the precise definition of platform varies by language.

So for example, I would like something like the following:

/frameworks/php/plain (or /frameworks/php/vanilla) /frameworks/php/laravel /frameworks/java/servlet /frameworks/java/netty /frameworks/java/spark /frameworks/java/wicket

I think this would be awesome. Thoughts?

— Reply to this email directly or view it on GitHub https://github.com/TechEmpower/FrameworkBenchmarks/issues/448#issuecomment-47945118 .

hamiltont commented 10 years ago

@jyentechempower I think closing this is a bit premature - we should ideally wait until a directory restructure has been completed and tested. #915 make doing this a lot simpler, because you can start using variables instead of hard directory positions, but it doesn't actually restructure the folder locations at all

hamiltont commented 10 years ago

Some thoughts on this:

ps - @fernandoacorreia were you ever able to complete/merge your work on automating the deployment? That would be amazing...

msmith-techempower commented 10 years ago

@hamiltont

With the current setup, it's entirely possible that framework A runs fine on a clean VM, but does not run at all if framework B is installed first

This is terrifying - can you give an example where this is true? I expect, apparently incorrectly, that each test is sandboxed with regard to everything except its dependencies (like nginx, java, etc).

hamiltont commented 10 years ago

@msmith-techempower Given that you excepted dependencies, I'll give a contrived example for a potential problem that is not dependency (meaning some script we run under /toolset, not dependency in the broader sense) related. I don't have an example case where this happens currently, but that might just be because these issues would be very hard to spot - they would just look like a normal test failure, which we have a lot of at the moment.

contrived example : mojolicious currently requires the perl module Mango version 0.30. If a new test is added (mojo-orm or something) that requries mango 0.9, it's install.sh will install 0.9. The install.sh for mojolicious wouldn't roll back the mango version, it would just think everything is ok because a newer version is installed, and bam failure when you run mojolicious. Disclaimer: I don't think this is actually possible for mojolicious or our other perl modules, because @jberger setup some nice perl module management stuff that I believe can declare "I specifically need mango 0.30"

bonus round : You may have noticed me complaining in general about installing stuff to the local system instead of installing stuff to a separate folder. There's a good reason, this causes potential for the exact same error as above, but for our dependency scripts. Pop up an ubuntu 12 box and run fw_depends mono nginx and watch the build fail with gcc: internal compiler error. Fun to do! But yea....bad for running our suite.

To completely invalidate my sentence, we need (1) all of our dependency scripts to install stuff into folders, and we need (2) to use language-specific package management solutions (I say environment in some places because a few languages can even change things like environment variables on a project-by-project basis). (1) is pretty simple, I think it will happen naturally as we're closing PRs. (2) is also pretty simple, but it depends heavily on closing this issue. Without a clean directory organization, (2) is the path to brimstone and dragons as you start changing little bits of toolset/ all over the place to detect what package management solution you're using

hamiltont commented 10 years ago

organize the /frameworks directory by language first, and framework/platform second

This isn't supported yet, but it should be fairly simple to add in now. That PR was just getting too big to continue work on

bhauer commented 10 years ago

Awesome to have this one closed!