Many of the flags have been removed and incorporated into the tree so that they are always parsed. This improves performance since we can parse the existing HTML while going through nodes and reduces the number of flags needed to gather information.
These flags include:
-m (all emails are retrieved whenever a tree is created)
-p (all phone numbers are retrieved whenever a tree is created)
c and cAll (classification is always done for all nodes)
Some flags that were not previously being used have been removed.
These flags include:
e
Some flags were redundant or not very useful.
These flags include:
--download (data can be saved as a JSON file or text tree with the --save flag), other formats can be added to this flag.
--gather (this is basically download and has been replaced by --save as well)
Some new flags have been added.
Includes:
v (turns on verbose logging which sets the logger to debug level)
disable-socks5 (constructs HTTP client without SOCKS5 proxy)
gotor has been completely removed and replaced with httpx.
None of the tests were working or actually being run, a new test suite needs to be developed. The code has been written in a more modular fashion which should help with this task.
Most of the code now has types added to the function definitions.
The version is now read directly from the pyproject.toml so it will update as the project gets updated using poetry.
Multiple arguments can be used in tandem such as save, visualize or info now.
This pull request contains some major changes.
Many of the flags have been removed and incorporated into the tree so that they are always parsed. This improves performance since we can parse the existing HTML while going through nodes and reduces the number of flags needed to gather information. These flags include:
-m(all emails are retrieved whenever a tree is created)-p(all phone numbers are retrieved whenever a tree is created)candcAll(classification is always done for all nodes)Some flags that were not previously being used have been removed. These flags include:
eSome flags were redundant or not very useful. These flags include:
--download(data can be saved as a JSON file or text tree with the--saveflag), other formats can be added to this flag.--gather(this is basically download and has been replaced by--saveas well)Some new flags have been added. Includes:
v(turns on verbose logging which sets the logger todebuglevel)disable-socks5(constructs HTTP client without SOCKS5 proxy)gotorhas been completely removed and replaced withhttpx.None of the tests were working or actually being run, a new test suite needs to be developed. The code has been written in a more modular fashion which should help with this task.
Most of the code now has types added to the function definitions.
The version is now read directly from the
pyproject.tomlso it will update as the project gets updated usingpoetry.Multiple arguments can be used in tandem such as save, visualize or info now.