search

Arachni / arachni

Web Application Security Scanner Framework
http://www.arachni-scanner.com
Other
3.7k stars 756 forks source link

distro friendly: ~/.arachni for all default output #474

Closed blshkv closed 9 years ago

blshkv commented 9 years ago

A similar issue has been fixed in the sqlmap recently: https://github.com/sqlmapproject/sqlmap/issues/746

Currently, arachni writes all output to the current system directory by default. If the tool is installed into /usr/share/aracyhni, for example, it would require a root permission so it could have "write" permissions.This is against Linux policy.

Please use ~/.arachni as an output directory by default. It should be the first priority for configuration files as well to overwrite any system settings.

ps. As a side effect, it should be possible to run the tool using a full path, not from a current directly only

Thank you.

Zapotek commented 9 years ago

You can set the ARACHNI_FRAMEWORK_LOGDIR and ARACHNI_WEBUI_LOGDIR env variables to change the default log directories when running from source. When running the packages, you can adjust these values in the system/environment file.

Also, can you elaborate on running the Arachni using a full path? I'm not sure what you mean.

Cheers

blshkv commented 9 years ago

that could be an option, I'll have a look at it. Thanks.

For the full path, here is the output:

blshkv@pt ~ $ /home/blshkv/arachni-1.0/bin/arachni
/home/blshkv/arachni-1.0/lib/arachni/state/audit.rb:22:in `<class:Audit>': uninitialized constant Arachni::State::Audit::Forwardable (NameError)
        from /home/blshkv/arachni-1.0/lib/arachni/state/audit.rb:21:in `<class:State>'
        from /home/blshkv/arachni-1.0/lib/arachni/state/audit.rb:10:in `<module:Arachni>'
        from /home/blshkv/arachni-1.0/lib/arachni/state/audit.rb:9:in `<top (required)>'
        from /home/blshkv/arachni-1.0/lib/arachni/state.rb:26:in `require_relative'
        from /home/blshkv/arachni-1.0/lib/arachni/state.rb:26:in `<class:State>'
        from /home/blshkv/arachni-1.0/lib/arachni/state.rb:14:in `<module:Arachni>'
        from /home/blshkv/arachni-1.0/lib/arachni/state.rb:9:in `<top (required)>'
        from /home/blshkv/arachni-1.0/lib/arachni/snapshot.rb:13:in `require_relative'
        from /home/blshkv/arachni-1.0/lib/arachni/snapshot.rb:13:in `<top (required)>'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /home/blshkv/arachni-1.0/lib/arachni/framework.rb:33:in `<module:Arachni>'
        from /home/blshkv/arachni-1.0/lib/arachni/framework.rb:20:in `<top (required)>'
        from /home/blshkv/arachni-1.0/lib/arachni.rb:56:in `require_relative'
        from /home/blshkv/arachni-1.0/lib/arachni.rb:56:in `<top (required)>'
        from /home/blshkv/arachni-1.0/ui/cli/framework.rb:9:in `require_relative'
        from /home/blshkv/arachni-1.0/ui/cli/framework.rb:9:in `<top (required)>'
        from /home/blshkv/arachni-1.0/bin/arachni:10:in `require_relative'
        from /home/blshkv/arachni-1.0/bin/arachni:10:in `<main>'

it was a different problem with the previous 0.4.7 version:

bash$ /usr/lib/arachni/bin/arachni
/usr/lib64/ruby/gems/1.9.1/gems/ethon-0.7.0/lib/ethon/multi.rb:97:in `block in set_attributes': The option: method is invalid. (Ethon::Errors::InvalidOption)
        from /usr/lib64/ruby/gems/1.9.1/gems/ethon-0.7.0/lib/ethon/multi.rb:95:in `each_pair'
        from /usr/lib64/ruby/gems/1.9.1/gems/ethon-0.7.0/lib/ethon/multi.rb:95:in `set_attributes'
        from /usr/lib64/ruby/gems/1.9.1/gems/ethon-0.7.0/lib/ethon/multi.rb:80:in `initialize'
        from /usr/lib64/ruby/gems/1.9.1/gems/typhoeus-0.6.8/lib/typhoeus/hydra.rb:92:in `new'
        from /usr/lib64/ruby/gems/1.9.1/gems/typhoeus-0.6.8/lib/typhoeus/hydra.rb:92:in `initialize'
        from /usr/lib64/arachni/lib/arachni/http.rb:115:in `new'
        from /usr/lib64/arachni/lib/arachni/http.rb:115:in `reset'
        from /usr/lib64/arachni/lib/arachni/http.rb:92:in `initialize'
        from /usr/lib64/ruby/1.9.1/singleton.rb:141:in `new'
        from /usr/lib64/ruby/1.9.1/singleton.rb:141:in `block in instance'
        from <internal:prelude>:10:in `synchronize'
        from /usr/lib64/ruby/1.9.1/singleton.rb:139:in `instance'
        from /usr/lib64/arachni/lib/arachni/framework.rb:146:in `initialize'
        from /usr/lib64/arachni/lib/arachni/ui/cli/cli.rb:74:in `new'
        from /usr/lib64/arachni/lib/arachni/ui/cli/cli.rb:74:in `initialize'
        from /usr/lib/arachni/bin/arachni:24:in `new'
        from /usr/lib/arachni/bin/arachni:24:in `<main>'
Zapotek commented 9 years ago

Ah, the fact that you're running it that way is coincidental, that error has a different cause which has been fixed. However, I couldn't reproduce it myself so could you retry with the nightlies and let me know if the error goes away?

blshkv commented 9 years ago

just tried the latest git, the same problem.

as for the variables, I can't specify $HOME, can I? Each user should have it's own ~.arachni dir.

Zapotek commented 9 years ago

Did you grab the master or the experimental branch? The fix is in experimental. Did you try using $HOME but it didn't work?

blshkv commented 9 years ago 
/home/blshkv/Downloads/arachni-experimental/bin/arachni
/usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require': cannot load such file -- childprocess (LoadError)
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser.rb:9:in `<top (required)>'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser_cluster/worker.rb:12:in `<module:Arachni>'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser_cluster/worker.rb:10:in `<top (required)>'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser_cluster.rb:44:in `<class:BrowserCluster>'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser_cluster.rb:16:in `<module:Arachni>'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/browser_cluster.rb:11:in `<top (required)>'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/framework.rb:42:in `<module:Arachni>'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni/framework.rb:20:in `<top (required)>'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni.rb:56:in `require_relative'
        from /home/blshkv/Downloads/arachni-experimental/lib/arachni.rb:56:in `<top (required)>'
        from /home/blshkv/Downloads/arachni-experimental/ui/cli/framework.rb:9:in `require_relative'
        from /home/blshkv/Downloads/arachni-experimental/ui/cli/framework.rb:9:in `<top (required)>'
        from /home/blshkv/Downloads/arachni-experimental/bin/arachni:10:in `require_relative'
        from /home/blshkv/Downloads/arachni-experimental/bin/arachni:10:in `<main>'
Zapotek commented 9 years ago

Did you run bundle install? If you're not familiar with Ruby's ecosystem I suggest you use the self-contained packages. In this case the nightlies, as they are built from the experimental branch.

blshkv commented 9 years ago

oh, more deps. Ok, I can't install some of them:

Installing stackprof (0.2.7) 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

    /usr/bin/ruby19 extconf.rb 
checking for rb_postponed_job_register_one()... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/bin/ruby19
extconf.rb:8:in `<main>': missing API: are you using ruby 2.1+? (RuntimeError)

Gem files will remain installed in /home/blshkv/.gem/ruby/1.9.1/gems/stackprof-0.2.7 for inspection.
Results logged to /home/blshkv/.gem/ruby/1.9.1/gems/stackprof-0.2.7/ext/gem_make.out

An error occurred while installing stackprof (0.2.7), and Bundler cannot continue.
Make sure that `gem install stackprof -v '0.2.7'` succeeds before bundling.
Zapotek commented 9 years ago

Try: bundle install --without prof

blshkv commented 9 years ago

ok, it is fixed in the experimental.

Now, back to the env variable... I can't specify a global variable with $HOME because it will propagate the current user (root) into this variable.

I can't find system/environment either. Where is the best location to specify it so an each new logged user will get the correct path?..

Zapotek commented 9 years ago

Like I said before, you better use the self-contained packages and set the locations in the system/environment file. That file is loaded during runtime and $HOME will be resolved to the current user's home directory.

You can also place any boot-up checks there as well, like creating $HOME/.arachni-<version> if it doesn't already exist. The full version can be found in the VERSION file.

blshkv commented 9 years ago

I had a look at standalone setup. Sounds way too complicated and not exactly what I want.

Can you add an option to use a current $HOME instead? I highlighted the sqlmap implementation in the first post. That would be perfect

Zapotek commented 9 years ago

I don't think that we're talking about the same thing. You just need to download one of the packages and make a 3 line change to system/environment, that's it.

blshkv commented 9 years ago

yes, I do not understand what you are talking about. Which package? How to download? I can't find system/environment

Zapotek commented 9 years ago

Latest stable release: http://www.arachni-scanner.com/download/ Nightly builds (built from the experimental branch): http://downloads.arachni-scanner.com/nightlies/

blshkv commented 9 years ago

Oh, ok. I'm actually porting it for Pentoo Linux which is basically Gentoo. So we don't use your precompiled packages and install from source. That's why I need a proper solution, not a workaround.

Zapotek commented 9 years ago

That is the proper solution, you are provided with the option to do what you want either way. Since you're working from source, why not just set the env variables in .bash_profile or .bashrc?

Am I missing something here?

blshkv commented 9 years ago

The subject says "distro friendly" ;-) So here the Goal:

That's an orthodox way of doing things and that's how sqlmap changed it. As a package maintainer, I can change any settings during a setup, but I can not ask each user to modify his .bashrc for each tool. I mean it will probably work, but that's not the usual way.

Zapotek commented 9 years ago

Fair enough, although I can't change the default behavior, not in the codebase, which is why I provide the packages. You'll either have to wrap the executables in shell scripts that set these env variables or just give up on Arachni.

Sorry man.

blshkv commented 9 years ago

a binary wrapper could be an option. Thanks for the hint

Zapotek commented 9 years ago

Just to be clear, I believe that building from source is a very bad idea. Projects provide packages because there are dependency issues from time to time and that's a way to make sure that everyone gets the same tested and stable environment, and you are completely bypassing that.

blshkv commented 9 years ago

Well, that's a very rhetorical topic.

I would agree with you if you would have ONE tool in your system. However, we have few hundreds. We can't afford installing 113Mb of precompiled package with all deps when the actually size of the software is 239Kb only. In fact, it is quite opposite: in a long run, the dependency issues become hell.

So building from source is THE ONLY way to go for us and that's why we like Gentoo. In this way we unbundle all dependencies and shared libraries and force tools to use system-wide available deps (tested and trusted).

So please make sure that all deps are specified properly in the Gemfile/Rakefile files. And we will build the same environment as you want it. Just much cleaner from Gentoo point of view.

Zapotek commented 9 years ago

Yeah you're right, this was an FYI, in case you get any issues the first thing to check should be the dependencies. As for installing from source, see: https://github.com/Arachni/arachni/wiki/Installation#source

blshkv commented 9 years ago

My bad, could you revisit this issue again. Basically, we need a variable for all output (configuration, reports, snapshots etc) by default, not only "logs".

I set both LOGDIR variables, and getting the following error:

 [*] Dumping audit results in '2014-09-21 14.28.00 +0800.afr'.
 [-] Permission denied - 2014-09-21 14.28.00 +0800.afr
/usr/lib64/arachni/lib/arachni/ui/cli/output.rb:79:in `initialize': Permission denied - error.log (Errno::EACCES)
        from /usr/lib64/arachni/lib/arachni/ui/cli/output.rb:79:in `open'
        from /usr/lib64/arachni/lib/arachni/ui/cli/output.rb:79:in `log_error'
        from /usr/lib64/arachni/lib/arachni/ui/cli/output.rb:75:in `print_error'
        from /usr/lib64/arachni/lib/arachni/ui/cli/cli.rb:124:in `rescue in run'
        from /usr/lib64/arachni/lib/arachni/ui/cli/cli.rb:98:in `run'
        from ./bin/arachni:24:in `<main>'

I guess, something like OUTPUT_PREFIX variable which would be a path of the tool by default (since you are not ready to make it distro friendly)

Zapotek commented 9 years ago

When running a CLI scan the error.log file and generated reports and snapshots are saved to the current working dir. Alternative locations for reports and snapshots can be set by the user as CLI parameters.

Different interfaces have different properties, you can't globally instruct Arachni to only write everything to a single directory. The env variables I've mentioned only set the location of the RPC and WebUI server logs, since these need to be grouped together by default.

As for this issue, are you expecting users to run applications from directories to which they don't have write permissions?

blshkv commented 9 years ago

Hm... I'm not using that experimental branch where the bug is fixed, so the current wrapper has to do the following:

cd /usr/lib/arachni
./bin/arachni

which makes the current directly always /usr/lib/arachni... I guess I have to wait for the next release.

But anyways, writing to the current directory is the bad habit since a user didn't specify any output and don't expect banch of files to be created. So he could be in any directory, type "arachni" (which is in system path in my case) and the tool would unexpectedly pollute his current directory with these files.

That's why I suggested to use ~/.arachni by default, which user can always explicitly set to a different location.

Zapotek commented 9 years ago

The whole reason for running a CLI scan is to get the report at the end, writing the product of the scan to the working directory (by default) is fully expected and proper behaviour.

As for the error.log file, that's written to the working dir to draw attention to itself, that could be considered slight pollution since its location is hardcoded, but it encourages people to review it and report errors so I'm fine with that.

We can debate this all day, but I like the current workflow and I won't be changing it.

blshkv commented 9 years ago

I just dont know any other tool which does that, that's why it is unexpected. Will probably need to write a patch and maintain it across all next versions which is .. annoying :\

But hey, thanks for the quick responses.

ZeroChaos- commented 9 years ago

This bug seems to have a lot of issues all merged together as one, oh how I love ruby stuff.

My real question is, why do you not want distros to be able to package your software? If you always store your output to the dir where arachni is installed then it can never be installed systemwide and no distro can package it. Plus writing everything in some random unpacked directory means that every time I upgrade I have to lose everything because I grabbed a new tarball and now things are in a new directory.

It would be far more sane to write out to ~/.arachni so that not only can non-root users run it, but the config files are in a sane place AND persist across upgrades.

The only downside of this is if you break the format of the files across versions, but if you aren't doing that then this way is much better for all concerned.

Zapotek commented 9 years ago

I've already mentioned the fact that you can specify alternative directories for the RPC logs.

  1. For the Framework, the RPC logs are the only output that is stored within the installation dir, so that should take care of your problems. Also, there is nothing to lose because, again, nothing else is written there, there are no default config files in the way you're thinking.
  2. For the WebUI (don't know if you're interested in packaging that too), you can still change the log location via an env variable and set the SQLite3 DB location via a Rails config file. That'll allow you to get the experience you want, with the WebUI's settings and data persisting across multiple versions -- you'll have to handle upgrading the DB though when upgrading Arachni, I'll show you how if it comes to that, it's simple enough.

Is there a reason why setting the environment variables is a problem for you? If so, then I can out these default log locations in a file and you can set the values when you package Arachni.

As for your comments on the self-contained packages, that is exactly what I want. I want them to not touch anything system related, I want multiple versions to co-exist, I want the user to explicitly decide whether to upgrade and from where, by giving the new version the location of the previous package, and I want them to be portable between machines, data and all.

You guys seems to think that I downed a fifth of scotch and then started chugging out code, I didn't, that workflow works for this project and that's my highest priority. And even though I'm not willing to change default behavior, I am willing to accommodate more workflows, like yours, so lets quit the lecturing and tell me what's the problem with the options you have available for setting write directories and I'll see what I can do.

ZeroChaos- commented 9 years ago

I agree, scotch isn't a great choice, I'd prefer a nice whiskey.

Having the locations set in a file would definitely be helpful, things like that are typically easy to sed at install time.

Zapotek commented 9 years ago

Will cf62e2e work for you? Shall I add something similar to the WebUI too?

blshkv commented 9 years ago

I don't really see a big change. The env variable was fine. It was easy to redefine the location by exporting DIRLOG=$HOME/.arachni/.

Could you introduce few more vars for all other files as well?

Zapotek commented 9 years ago

Just to keep things clean, can we first decide on what's needed so that I can only implement that and not have a bunch of useless features laying around? Are we set on using env variables? Shall I revert the last commit and introduce variables for the other files?

I actually think the env variable approach is better since it allows for paths to be evaluated during runtime.

blshkv commented 9 years ago

It is a config file usually where it is possible to specify a path for each file. But please make sure it is possible to specify a home directory (using '~' or '$HOME' variables?). So I'm ok with any option, as long as it works ;-) Thanks a lot

Zapotek commented 9 years ago

Cool, config file it is then, I'll get to this tomorrow.

ZeroChaos- commented 9 years ago

yes config file please. When installing something distro-wide if you use envvars then they must always be set which makes life get uglier and uglier. the config file can be set at install time and even edited by the user if desired.

necrose99 commented 9 years ago

blshkv, ZeroChaos , when they speak, Defcon,Blackhat etc grows quiet,

the lite weight pentest repo we run most of the packages are built from the stock Pentoo ,and wrapped for lazy or more student or novice Sabayon users, but all in common :Gentoo: Sabayon aims for simplicity of a bin repo with DB (and a gui package manager for more novice users) and Redhat like Installs of Gentoo, however the full power of gentoo can be unleashed immediately. {since my laptop has hardware the stock Gentoo installer goes berserk at..} ours is quite bleeding edge, if we punch mach 5 and the packages break , or via digging for new packages or find an old one that hasnt been updated to mach upstream , we gladly share to the Pentoo team. arachni should be able to install no matter the distro, no matter the OS. although a pain , true o/s portability is a good goal to reach for.

blshkv, ZeroChaos , when they speak, agree or disagree , pray'tell you should wisely listen.

Zapotek commented 9 years ago

Holy crap you guys have ganged up on me. :P

As for OS portability, feel free to blame the MRI team, the Framework runs on Windows but Ruby crashes. I've gotten promising results with JRuby, which will eventually let Arachni run on Windows.

Anyhow, point taken, the issue will be resolved.

ZeroChaos- commented 9 years ago

Sorry, people follow us around and comment as they wish. This is what happens when you have users ;-) In this case you seem to already be helping us so I don't think the additional kick is needed. I'd prefer that you have defaults more in line with what we are suggesting, you are knowingly going another way (one which limits your users imho) but that's your choice.

That said, Anton is the one packaging arachni, and if he is happy then I am happy, and since Necrose is using our stuff, if I'm happy he will be happy :-)

Zapotek commented 9 years ago

This should do the trick, let me know if I've missed anything.

blshkv commented 9 years ago

Perfect, seems working!

thanks a bunch

Zapotek commented 9 years ago

No problem. Let me know if you're interested in packaging the WebUI too, Rails has its own conventions.

blshkv commented 9 years ago

Yes, we will definitely include webUI at some point too. But I'm happy with cli for now. It runs and generates reports. In the right place now ;-)

Zapotek commented 9 years ago

One day I'll find something that's not kosher on Pentoo and hire a skywriter...

ZeroChaos- commented 9 years ago

skywriter is fine, just make sure to use the bug tracker too ;-)