search

Bee-Mar / mmpm

MagicMirror Package Manager
MIT License
198 stars 30 forks source link

BUG: UI is accessible, active modules work, but marketplace is empty and config files are not loading #163

Closed KKdev15 closed 9 months ago

KKdev15 commented 9 months ago

Describe the Bug

Is this an issue with the CLI or GUI? GUI

Is this something that should be put in the Wiki? NO

A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Open MMPM GUI

Expected behavior I expect to see modules that are available for install in Marketplace and the config files in the Config Editor

Screenshots image image

Operating System, Python3 Version, and Web Browser\ (Please complete the following information)

MMPM Environment Settings

MMPM Log Files

MagicMirror Config

 *
 * By Michael Teeuw https://michaelteeuw.nl
 * MIT Licensed.
 *
 * For more information on how you can configure this file
 * see https://docs.magicmirror.builders/configuration/introduction.html
 * and https://docs.magicmirror.builders/modules/configuration.html
 *
 * You can use environment variables using a `config.js.template` file instead of `config.js`
 * which will be converted to `config.js` while starting. For more information
 * see https://docs.magicmirror.builders/configuration/introduction.html#enviromnent-variables
 */
let config = {
    address: "localhost",   // Address to listen on, can be:
                            // - "localhost", "127.0.0.1", "::1" to listen on loopback interface
                            // - another specific IPv4/6 to listen on a specific interface
                            // - "0.0.0.0", "::" to listen on any interface
                            // Default, when address config is left out or empty, is "localhost"
    port: 8080,
    basePath: "/",  // The URL path where MagicMirror² is hosted. If you are using a Reverse proxy
                                    // you must set the sub path here. basePath must end with a /
    ipWhitelist: [],    // Set [] to allow all IP addresses
                                    // or add a specific IPv4 of 192.168.1.5 :
                                    // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
                                    // or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
                                    // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

    useHttps: false,            // Support HTTPS or not, default "false" will use HTTP
    httpsPrivateKey: "",    // HTTPS private key path, only require when useHttps is true
    httpsCertificate: "",   // HTTPS Certificate path, only require when useHttps is true

    language: "de",
    locale: "de-DE",
    logLevel: ["INFO", "LOG", "WARN", "ERROR"], // Add "DEBUG" for even more logging
    timeFormat: 24,
    units: "metric",

    modules: [
/* some modules... */
{
    module: "MMM-mmpm"
},
    ]
};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") { module.exports = config; }

Additional Information From the error shown in the Screenshots, I assumed, that the error war part of the mmpm.api process started by pm2. Therefore I checked the logs and found the following error repeating constantly:

[2024-01-14 18:47:27 +0100] [2439] [ERROR] Worker (pid:2440) exited with code 3
[2024-01-14 18:47:27 +0100] [2439] [ERROR] Shutting down: Master
[2024-01-14 18:47:27 +0100] [2439] [ERROR] Reason: Worker failed to boot.
[2024-01-14 18:47:28 +0100] [2441] [INFO] Starting gunicorn 21.2.0
[2024-01-14 18:47:28 +0100] [2441] [INFO] Listening at: http://0.0.0.0:7891 (2441)
[2024-01-14 18:47:28 +0100] [2441] [INFO] Using worker: gevent
[2024-01-14 18:47:28 +0100] [2442] [INFO] Booting worker with pid: 2442
[2024-01-14 18:47:32 +0100] [2442] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 147, in init_process
    super().init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/wsgi.py", line 28, in <module>
    from mmpm.api.entrypoint import app
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/api/entrypoint.py", line 8, in <module>
    from flask import Flask, Response
  File "/home/pi/.local/lib/python3.9/site-packages/flask/__init__.py", line 14, in <module>
    from jinja2 import escape
ImportError: cannot import name 'escape' from 'jinja2' (/home/pi/.local/lib/python3.9/site-packages/jinja2/__init__.py)

From this Stackoverflow issue, I figured out, that maybe the used version of Flask and jinja don't match: https://stackoverflow.com/questions/71718167/importerror-cannot-import-name-escape-from-jinja2

Is this a general dependency problem in mmpm or is this just on my Raspi? If it is just on my Raspi, which version of Flask and jinja do I need and how do I install them into the correct place? Are Flask and jinja installed globally?

Thanks!

EDIT These are the moduled installed on my Raspi (Flask = 1.1.2 and Jinja2 = 3.1.3):

pi@smartmirror:~ $ python3 -m pip list
Package            Version
------------------ ---------
arandr             0.1.10
argcomplete        3.2.1
astroid            2.5.1
asttokens          2.0.4
automationhat      0.2.0
beautifulsoup4     4.12.2
bidict             0.22.1
blinker            1.7.0
blinkt             0.1.2
buttonshim         0.0.2
Cap1xxx            0.1.3
certifi            2020.6.20
chardet            4.0.0
charset-normalizer 3.3.2
click              8.1.7
colorama           0.4.4
colorzero          1.1
cryptography       3.3.2
cupshelpers        1.0
dbus-python        1.2.16
distro             1.5.0
docutils           0.16
drumhat            0.1.0
envirophat         1.0.0
ExplorerHAT        0.4.2
Flask              1.1.2
Flask-Cors         4.0.0
Flask-SocketIO     5.3.6
fourletterphat     0.1.0
gevent             23.9.1
gevent-websocket   0.10.1
gitdb              4.0.11
GitPython          3.1.41
gpiozero           1.6.2
greenlet           3.0.3
gunicorn           21.2.0
h11                0.14.0
html5lib           1.1
idna               2.10
importlib-metadata 7.0.1
isort              5.6.4
itsdangerous       2.1.2
jedi               0.18.0
Jinja2             3.1.3
lazy-object-proxy  0.0.0
logilab-common     1.8.1
lxml               4.6.3
MarkupSafe         2.1.3
mccabe             0.6.1
microdotphat       0.2.1
mmpm               4.1.0
mote               0.0.4
motephat           0.0.3
mypy               0.812
mypy-extensions    0.4.3
numpy              1.19.5
oauthlib           3.1.0
packaging          23.2
pantilthat         0.0.7
parso              0.8.1
pexpect            4.8.0
pgzero             1.2
phatbeat           0.1.1
pianohat           0.1.0
picamera           1.13
picamera2          0.3.12
pidng              4.0.9
piexif             1.1.3
piglow             1.2.5
pigpio             1.78
Pillow             8.1.2
pip                23.3.2
prompt-toolkit     3.0.43
psutil             5.8.0
pycairo            1.16.2
pycups             2.0.1
pygame             1.9.6
Pygments           2.17.2
PyGObject          3.38.0
pyinotify          0.9.6
PyJWT              1.7.1
pylint             2.7.2
PyOpenGL           3.1.5
pyOpenSSL          20.0.1
PyQt5              5.15.2
PyQt5-sip          12.8.1
pyserial           3.5b0
pysmbc             1.0.23
python-apt         2.2.1
python-engineio    4.8.2
python-prctl       1.7
python-socketio    5.11.0
rainbowhat         0.1.0
reportlab          3.5.59
requests           2.31.0
requests-oauthlib  1.0.0
responses          0.12.1
roman              2.0.0
RPi.GPIO           0.7.0
RTIMULib           7.2.1
scrollphat         0.0.7
scrollphathd       1.2.1
Send2Trash         1.6.0b1
sense-hat          2.6.0
setuptools         52.0.0
simple-websocket   1.0.0
simplejpeg         1.6.4
simplejson         3.17.2
six                1.16.0
skywriter          0.0.7
smmap              5.0.1
sn3218             1.2.7
soupsieve          2.2.1
spidev             3.5
ssh-import-id      5.10
termcolor          2.4.0
thonny             4.0.1
toml               0.10.1
touchphat          0.0.1
twython            3.8.2
typed-ast          1.4.2
typing-extensions  3.7.4.3
unicornhathd       0.0.4
urllib3            1.26.5
v4l2-python3       0.3.2
wcwidth            0.2.13
webencodings       0.5.1
websocket-client   1.7.0
Werkzeug           1.0.1
wheel              0.34.2
wrapt              1.12.1
wsproto            1.2.0
yaspin             3.0.1
zipp               3.17.0
zope.event         5.0
zope.interface     6.1
KKdev15 commented 9 months ago

I was able to solve the bug.

Install Jinja==3.1.0 and flask==2.2.0 As reported by pip when installing an old version of Jinja:

mmpm 4.1.0 requires flask>=2.2.0, but you have flask 1.1.2 which is incompatible.
mmpm 4.1.0 requires jinja2>=3.1.0, but you have jinja2 3.0.3 which is incompatible.

Shell commands:

python3 -m pip install Jinja2==3.1.0
python3 -m pip install flask==2.2.0

After this I got another error in the mmpm.api logs:

[2024-01-14 19:53:33 +0100] [1630] [INFO] Worker exiting (pid: 1630)
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Worker (pid:1630) exited with code 3
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Shutting down: Master
[2024-01-14 19:53:33 +0100] [1629] [ERROR] Reason: Worker failed to boot.
[2024-01-14 19:53:34 +0100] [1631] [INFO] Starting gunicorn 21.2.0
[2024-01-14 19:53:34 +0100] [1631] [INFO] Listening at: http://0.0.0.0:7891 (1631)
[2024-01-14 19:53:34 +0100] [1631] [INFO] Using worker: gevent
[2024-01-14 19:53:34 +0100] [1632] [INFO] Booting worker with pid: 1632
[2024-01-14 19:53:38 +0100] [1632] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 147, in init_process
    super().init_process()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/pi/.local/lib/python3.9/site-packages/gunicorn/util.py", line 371, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/wsgi.py", line 28, in <module>
    from mmpm.api.entrypoint import app
  File "/home/pi/.local/lib/python3.9/site-packages/mmpm/api/entrypoint.py", line 8, in <module>
    from flask import Flask, Response
  File "/home/pi/.local/lib/python3.9/site-packages/flask/__init__.py", line 5, in <module>
    from .app import Flask as Flask
  File "/home/pi/.local/lib/python3.9/site-packages/flask/app.py", line 29, in <module>
    from werkzeug.urls import url_quote
ImportError: cannot import name 'url_quote' from 'werkzeug.urls' (/home/pi/.local/lib/python3.9/site-packages/werkzeug/urls.py)

I could solve this by installing an old version of Werkzeug: https://stackoverflow.com/questions/77213053/why-did-flask-start-failing-with-importerror-cannot-import-name-url-quote-fr python3 -m pip install Werkzeug==2.2.2

KKdev15 commented 9 months ago

Maybe it would make sense to specify the package-dependencies more strictly in the requirements.txt of the mmpm Python module. This way incompatibilities like that should not be a problem.

Bee-Mar commented 9 months ago

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

Bee-Mar commented 9 months ago

@KKdev15 also, I'm unsure of how you have flask version 1.1.2 installed, with MMPM operating considering the dependencies within the pyproject.toml are:

dependencies = [
  "Flask-Cors>=3.0.8",
  "argcomplete>=2.0.0",
  "argparse>=1.4",
  "beautifulsoup4>=4.11.1",
  "colorama>=0.4.3",
  "flask-socketio>=5.3.2",
  "flask>=2.2.0",
  "gevent>=23.9.1",
  "gunicorn>=20.1.0",
  "jinja2>=3.1.0",
  "pygments>=2.12.0",
  "python-socketio[client]>=5.7.0",
  "requests>=2.28.0",
  "yaspin>=2.3.0",
  "pip>=23.3.1",
  "gitpython>=3.1.40",
  "gevent-websocket>=0.10.1",
  "prompt-toolkit>=3.0.43",
  "importlib_resources; python_version<'3.9'",
]
Bee-Mar commented 9 months ago

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

@KKdev15 can you provide the version of mmpm you have installed?

KKdev15 commented 9 months ago

@KKdev15 what version of mmpm do you have installed? Are you on the latest 4.1.0?

@KKdev15 can you provide the version of mmpm you have installed?

@Bee-Mar I have installed the latest version (4.1.0) of mmpm.

I guess the problem was, that I also have the plugin MMM-FastNotes installed. It's dependencies (flask == 1.1.2) are conflicting with the dependencies of mmpm.

Anyway MMM-FastNotes seems to work with the mmpm dependencies as well...

A general question from my side: Is it possible to install different versions of the same package to make sure, that each package has exactly the dependencies it requires? I'm not a Node.js expert, but maybe there is something like venvs in Python?

Bee-Mar commented 9 months ago

@KKdev15 ah, well that's not something I expected. Yep, python has virtual environments. You could install mmpm using a virtualenv if you'd like. You'd basically do the following to get it setup:

python3 -m venv venv # creates a virtualenv named 'venv'
source venv/bin/activate
python3 -m pip install mmpm

You'd have run source venv/bin/activate prior to using the mmpm CLI, or you could alias it to make it easier. The installation of the UI will pickup the correct python path even if you use a virtualenv.

Another option is to use pipx https://github.com/pypa/pipx. That handles the virtualenv creation for you and makes an alias so you don't need to do anything special other than run the CLI like you normally would:

python3 -m pip install pipx
pipx install mmpm

Then just run mmpm from terminal.

Something else that could be interesting from my end is detecting if the module being installed has python requirements and prompting the user if they want to setup the module in a virtualenv and having mmpm handle that so things are isolated.

Bee-Mar commented 9 months ago

@KKdev15 one issue with that module is they made the requirements too strict. Requiring version numbers with == hamstrings anyone else installing the same dependency. It's typically a very bad idea to specify dependencies like that unless you're running the application in a container or environment you know won't conflict with anything else.

Bee-Mar commented 9 months ago

@KKdev15 in short, for simplicity sake, I'd recommend the following:

python3 -m pip uninstall mmpm
python3 -m pip install pipx
pipx install mmpm

Then go about things how you normally would. The should resolve the issue.

Bee-Mar commented 9 months ago

@KKdev15 I'm going to close this issue since the problem was related to another MagicMirror module. If you're still having issues feel free to respond in this thread or email me.

minkshaman commented 7 months ago

@Bee-Mar Hi, I've had the same issue after installing FastNotes, and I've followed your code block, but it hasn't helped!

do you have a suggestion?

minkshaman commented 7 months ago

Additional: HTML inspector shows CORS is being blocked.

How do I fix this?