search

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
http://www.spiderfoot.net
MIT License
13.1k stars 2.27k forks source link

2023-06-25 17:35:53,055 [CRITICAL] sf : Failed to load modules: cannot import name 'etree' from 'lxml' (/usr/lib/python3/dist-packages/lxml/__init__.py) #1797

Open MiChaelinzo opened 1 year ago

MiChaelinzo commented 1 year ago

Please provide a description of the issue and any relevant error messages. 2023-06-25 17:35:53,055 [CRITICAL] sf : Failed to load modules: cannot import name 'etree' from 'lxml' (/usr/lib/python3/dist-packages/lxml/init.py) Traceback (most recent call last): File "/home/michael/spiderfoot/./sf.py", line 144, in main sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py']) File "/home/michael/spiderfoot/spiderfoot/helpers.py", line 161, in loadModulesAsDict mod = import('modules.' + modName, globals(), locals(), [modName]) File "/home/michael/spiderfoot/modules/sfp_filemeta.py", line 19, in import docx File "/usr/lib/python3/dist-packages/docx/init.py", line 3, in from docx.api import Document # noqa File "/usr/lib/python3/dist-packages/docx/api.py", line 14, in from docx.package import Package File "/usr/lib/python3/dist-packages/docx/package.py", line 9, in from docx.opc.package import OpcPackage File "/usr/lib/python3/dist-packages/docx/opc/package.py", line 9, in from docx.opc.part import PartFactory File "/usr/lib/python3/dist-packages/docx/opc/part.py", line 12, in from .oxml import serialize_part_xml File "/usr/lib/python3/dist-packages/docx/opc/oxml.py", line 12, in from lxml import etree ImportError: cannot import name 'etree' from 'lxml' (/usr/lib/python3/dist-packages/lxml/init.py)

If you can provide an application stack trace that is even better.

What version of Python are you using? Python 3.10.7 What version of SpiderFoot are you using (stable release or Git master branch)? Git master branch You may also wish to check if your issue has been posted previously:

sam5epi0l commented 1 year ago 
pip uninstall lxml
pip install lxml

Reference

MiChaelinzo commented 1 year ago 
pip uninstall lxml
pip install lxml

Reference

michael@home:~/spiderfoot$ sudo pip uninstall lxml [sudo] password for michael: sudo: unable to execute /usr/local/bin/pip: No such file or directory

michael@home:~/spiderfoot$ pip3 uninstall lxml Found existing installation: lxml 4.9.2 Not uninstalling lxml at /usr/lib/python3/dist-packages, outside environment /usr Can't uninstall 'lxml'. No files were found to uninstall.

sam5epi0l commented 1 year ago

Maybe the package was installed with distro-package managers, if your distribution is based on Debian try:

sudo apt remove python3-lxml

or, try manually removing the package with:

sudo rm -rf /usr/lib/python3/dist-packages/lxml
MiChaelinzo commented 1 year ago

When I sudo apt remove python3-lxml.

 E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution). 
michael@home:~$ sudo rm -rf /usr/lib/python3/dist-packages/lxml
michael@home:~$ sudo pip3 install lxml
Requirement already satisfied: lxml in /usr/lib/python3/dist-packages (4.9.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
michael@home:~$ pip3 install lxml
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: lxml in /usr/lib/python3/dist-packages (4.9.2)

I tried sudo apt -fix--broken install 

ModuleNotFoundError: No module named 'apt_pkg'
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 818000 files and directories currently installed.)
Preparing to unpack .../python3-numpy_1%3a1.24.2-1_amd64.deb ...
Unpacking python3-numpy (1:1.24.2-1) over (1:1.21.5-1+b1) ...
dpkg: error processing archive /var/cache/apt/archives/python3-numpy_1%3a1.24.2-1_amd64.deb (--unpack):
 trying to overwrite '/usr/bin/f2py', which is also in package python-numpy 1:1.16.5-5
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/python3-numpy_1%3a1.24.2-1_amd64.deb
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
sam5epi0l commented 1 year ago

Do not use pip3 install with sudo ever.

We gotta do some trial and error here. Do sudo pip3 uninstall lxml Since numpy package is broken, remove it for now:

dpkg -P python3-numpy
dpkg -P python-numpy

Run sudo apt --fix-broken install Run sudo apt update && sudo apt install python3 python3-pip python3-lxml

MiChaelinzo commented 1 year ago 
Processing triggers for shared-mime-info (2.2-1) ...
Errors were encountered while processing:
 python3-lxml:amd64
 python3-fonttools
 python3-matplotlib
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
also 
sudo apt update && sudo apt install python3 python3-pip lxml
Hit:2 https://dl.google.com/linux/chrome/deb stable InRelease 
Hit:1 http://kali.download/kali kali-rolling InRelease
Reading package lists... Done   
Building dependency tree... Done
Reading state information... Done
2950 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package lxml

the code above messed up my fortune I used fortune for daily quotes logging in the lxdeterminal: ModuleNotFoundError: No module named 'fortune' I installed it again with pip3 

2023-07-02 17:05:35,657 [CRITICAL] sf : Failed to load modules: No module named 'lxml'
Traceback (most recent call last):
  File "/home/spiderfoot/./sf.py", line 144, in main
    sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spiderfoot/spiderfoot/helpers.py", line 161, in loadModulesAsDict
    mod = __import__('modules.' + modName, globals(), locals(), [modName])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/spiderfoot/modules/sfp_filemeta.py", line 19, in <module>
    import docx
  File "/usr/lib/python3/dist-packages/docx/__init__.py", line 3, in <module>
    from docx.api import Document  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docx/api.py", line 14, in <module>
    from docx.package import Package
  File "/usr/lib/python3/dist-packages/docx/package.py", line 9, in <module>
    from docx.opc.package import OpcPackage
  File "/usr/lib/python3/dist-packages/docx/opc/package.py", line 9, in <module>
    from docx.opc.part import PartFactory
  File "/usr/lib/python3/dist-packages/docx/opc/part.py", line 12, in <module>
    from .oxml import serialize_part_xml
  File "/usr/lib/python3/dist-packages/docx/opc/oxml.py", line 12, in <module>
    from lxml import etree
ModuleNotFoundError: No module named 'lxml'

lxml the only problem no etree modules error anymore

sam5epi0l commented 1 year ago

I forgot to add python3- before lxml in last command. I see you're using Kali Linux repo here. I've tested and it seems working with the following command:

sudo pip3 uninstall lxml && sudo rm -rf /usr/lib/python3/dist-packages/lxml
sudo apt update && sudo apt install python3 python3-pip python3-lxml
MiChaelinzo commented 1 year ago 
dpkg: error processing package python3-matplotlib (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.36-9) ...
Processing triggers for man-db (2.10.2-3) ...
Processing triggers for kali-menu (2022.4.1) ...
Errors were encountered while processing:
 python3-lxml:amd64
 python3-fonttools
 python3-matplotlib
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

I got this error 


sudo python3 ./sf.py -l 127.0.0.1:5009
2023-07-02 21:50:25,963 [CRITICAL] sf : Failed to load modules: No module named 'lxml'
Traceback (most recent call last):
  File "/home/michael/spiderfoot/./sf.py", line 144, in main
    sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/spiderfoot/spiderfoot/helpers.py", line 161, in loadModulesAsDict
    mod = __import__('modules.' + modName, globals(), locals(), [modName])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/spiderfoot/modules/sfp_filemeta.py", line 19, in <module>
    import docx
  File "/usr/lib/python3/dist-packages/docx/__init__.py", line 3, in <module>
    from docx.api import Document  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docx/api.py", line 14, in <module>
    from docx.package import Package
  File "/usr/lib/python3/dist-packages/docx/package.py", line 9, in <module>
    from docx.opc.package import OpcPackage
  File "/usr/lib/python3/dist-packages/docx/opc/package.py", line 9, in <module>
    from docx.opc.part import PartFactory
  File "/usr/lib/python3/dist-packages/docx/opc/part.py", line 12, in <module>
    from .oxml import serialize_part_xml
  File "/usr/lib/python3/dist-packages/docx/opc/oxml.py", line 12, in <module>
    from lxml import etree
ModuleNotFoundError: No module named 'lxml'

python3 ./sf.py -l 127.0.0.1:5009
2023-07-02 21:51:30,113 [CRITICAL] sf : Failed to load modules: No module named 'lxml'
Traceback (most recent call last):
  File "/home/michael/spiderfoot/./sf.py", line 144, in main
    sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/spiderfoot/spiderfoot/helpers.py", line 161, in loadModulesAsDict
    mod = __import__('modules.' + modName, globals(), locals(), [modName])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/michael/spiderfoot/modules/sfp_filemeta.py", line 19, in <module>
    import docx
  File "/usr/lib/python3/dist-packages/docx/__init__.py", line 3, in <module>
    from docx.api import Document  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/docx/api.py", line 14, in <module>
    from docx.package import Package
  File "/usr/lib/python3/dist-packages/docx/package.py", line 9, in <module>
    from docx.opc.package import OpcPackage
  File "/usr/lib/python3/dist-packages/docx/opc/package.py", line 9, in <module>
    from docx.opc.part import PartFactory
  File "/usr/lib/python3/dist-packages/docx/opc/part.py", line 12, in <module>
    from .oxml import serialize_part_xml
  File "/usr/lib/python3/dist-packages/docx/opc/oxml.py", line 12, in <module>
    from lxml import etree
ModuleNotFoundError: No module named 'lxml'
sam5epi0l commented 1 year ago

Try this out to fix dpkg:

sudo dpkg --configure -a
sudo apt-get -f install
sudo apt install python3-matplotlib --reinstall