search

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
http://www.spiderfoot.net
MIT License
13.28k stars 2.29k forks source link

create setup.py and support system wide installation #1176

Open blshkv opened 3 years ago

blshkv commented 3 years ago

This is a common problem that many tools which we want to distribute do not support system wide installation. Some of these tools provide their own "install.sh" or "setup.sh" or "requirements.txt" with a distro specific commands (sudo apt-install) to install dependencies instead of a proper setup.py. With a proper setup.py script, an end user can install it using the following single command: python3 setup.py install --user. It also a distro friendly and makes life for maintainers easier.

The end result, a FHS compliant directory structure should be created. In case if software produce any output, a local user directory should be used, such as ~.//

https://github.com/pentoo/pentoo-overlay/issues/596

blshkv commented 3 years ago

Please apply the following patch: https://github.com/pentoo/pentoo-overlay/blob/master/app-forensics/spiderfoot/files/spiderfoot-3.3_add_support_user_homedir.patch

bcoles commented 3 years ago

Please apply the following patch: https://github.com/pentoo/pentoo-overlay/blob/master/app-forensics/spiderfoot/files/spiderfoot-3.3_add_support_user_homedir.patch

There's a lot of work going on at the moment to refactor SpiderFoot as a package. Unfortunately that's still a long way off.

The patch above is going to break a bunch of things. But I agree something similar should be implemented. I'll notify you in this thread when complete.

blshkv commented 3 years ago

yeah, we adopted that patch a few times for each new version, but enough is enough ;-) I masked spiderfoot as "broken" until the refactoring is done, and we have a proper setup.py with user home support.

Thanks!

bcoles commented 3 years ago

yeah, we adopted that patch a few times for each new version, but enough is enough ;-) I masked spiderfoot as "broken" until the refactoring is done, and we have a proper setup.py with user home support.

Thanks!

As a stop gap measure, in theory you could force the path for the cache and database by setting the SPIDERFOOT_CACHE and SPIDERFOOT_DATA environment variables respectively. This will work for the cache, but as you suggest in https://github.com/pentoo/pentoo-overlay/issues/739 SPIDERFOOT_DATA is currently broken. This is a separate issue which will be fixed long before SpiderFoot is packaged.

bcoles commented 3 years ago

The database file is now loaded from $HOME/.spiderfoot/spiderfoot.db. Fixed on master branch in #1343.

Support for the SPIDERFOOT_DATA environment variable has also been fixed. The environment variable takes priority. The home directory is used if the environment variable is not set.

This should be sufficient for your use case and remove the need for your patches.

The next official release should not be far away. Unfortunately, it will not support installation with setup.py.

blshkv commented 3 years ago

https://github.com/pentoo/pentoo-overlay/issues/740 Please make sure that logs are also handled properly in the next version

bcoles commented 3 years ago

pentoo/pentoo-overlay#740 Please make sure that logs are also handled properly in the next version

The logs are now stored in $HOME/.spiderfoot/logs. Fixed on master branch in #1356.

The log directory may be overwritten by SPIDERFOOT_LOGS environment variable.

bcoles commented 3 years ago

@blshkv SpiderFoot 3.5 has been released. This should address the issues with the database, cache data, and logs. These are all now loaded/stored in ~/.spiderfoot/. Hopefully this is sufficient to remove the "broken" status from the Pentoo SpiderFoot port.

blshkv commented 3 years ago

Almost there. setup.py or pyproject.toml (PEP 517) are still missing.

ShutdownRepo commented 1 year ago

Any update on having a setup.py file? The requirements of the projects are very restrictive (since they are versioned), which creates conflicts with other tools. The great alternative to dockerised things is python virtual environments (like pipx for instance), and a setup.py file would be needed for pipx to be able to install spiderfoot

bcoles commented 1 year ago

Any update on having a setup.py file?

No. There has been no progress on creating a setup.py file. This is still desirable and the preferred approach for deployment.

Progress towards refactoring the libraries is ongoing at a glacial pace.

The requirements of the projects are very restrictive

The test requirements are restrictive and locked to specific versions; however, these are optional for development purposes only. They aren't required for installation and operation.

The required dependencies defined in requirements.txt are loosely versioned and not locked. They specify a range: a minimum version and sometimes specify a maximum version.

The vast majority of the dependencies (with a few inconvenient exceptions) use the latest major version branch. If these are causing conflicts with other software, the other software should be updated.

In instances where a minimum version is enforced, this is usually because older versions are not compatible or contain security vulnerabilities.

In instances where a maximum version is enforced, this is usually because newer versions are not compatible. Admittedly, we're a bit slow on checking whether the newest versions of dependencies are compatible. The maximum version, when set, may be safe to extend.