microsoft / Oryx

Build your repo automatically.
Other
752 stars 174 forks source link

Oryx build is successful but app crashes due to pip packages not being found #1968

Closed tjprescott closed 1 year ago

tjprescott commented 1 year ago

Bug Report

The application startup logs show that the pip packages are clearly not being found despite being successfully installed by Oryx.

Yes! You can see here that the build and deploy stages of Github actions are successful: image

Oryx Logs These are also successful as far as I can tell. The pip packages are clearly installed.

Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.10 -p virtualenv_name=antenv --log-file /tmp/build-debug.log  -i /tmp/8db4550c4a723fb --compress-destination-dir | tee /tmp/oryx-build.log
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues

Oryx Version: 0.2.20230210.1, Commit: a49c8f6b8abbe95b4356552c4c884dea7fd0d86e, ReleaseTagName: 20230210.1

Build Operation ID: 2c49d9c6b59f341b
Repository Commit : 1491df78-717e-4ad1-810a-bbfe0cc4d4cc
OS Type           : bullseye
Image Type        : githubactions

Detecting platforms...
Detected following platforms:
  python: 3.10.8

Using intermediate directory '/tmp/8db4550c4a723fb'.

Copying files to the intermediate directory...
Done in 1 sec(s).

Source directory     : /tmp/8db4550c4a723fb
Destination directory: /home/site/wwwroot

Python Version: /tmp/oryx/platforms/python/3.10.8/bin/python3.10
Creating directory for command manifest file if it does not exist
Removing existing manifest file
Python Virtual Environment: antenv
Creating virtual environment...
Activating virtual environment...
Running pip install...
[05:49:24+0000] Collecting black
[05:49:24+0000]   Using cached black-23.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)
[05:49:25+0000] Collecting dash
[05:49:25+0000]   Using cached dash-2.9.3-py3-none-any.whl (10.2 MB)
[05:49:25+0000] Collecting folium
[05:49:25+0000]   Using cached folium-0.14.0-py2.py3-none-any.whl (102 kB)
[05:49:25+0000] Collecting geojson
[05:49:25+0000]   Using cached geojson-3.0.1-py3-none-any.whl (15 kB)
[05:49:25+0000] Collecting geopy
[05:49:25+0000]   Using cached geopy-2.3.0-py3-none-any.whl (119 kB)
[05:49:25+0000] Collecting gpxpy
[05:49:25+0000]   Using cached gpxpy-1.5.0-py3-none-any.whl
[05:49:26+0000] Collecting mypy
[05:49:26+0000]   Using cached mypy-1.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
[05:49:28+0000] Collecting pandas
[05:49:28+0000]   Using cached pandas-2.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)
[05:49:28+0000] Collecting plotly
[05:49:29+0000]   Using cached plotly-5.14.1-py2.py3-none-any.whl (15.3 MB)
[05:49:30+0000] Collecting psycopg2
[05:49:30+0000]   Using cached psycopg2-2.9.6-cp310-cp310-linux_x86_64.whl
[05:49:30+0000] Collecting python-dotenv
[05:49:30+0000]   Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)
[05:49:30+0000] Collecting rdp
[05:49:30+0000]   Using cached rdp-0.8-py3-none-any.whl
[05:49:31+0000] Collecting scipy
[05:49:31+0000]   Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
[05:49:33+0000] Collecting shapely
[05:49:33+0000]   Using cached shapely-2.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
[05:49:33+0000] Collecting platformdirs>=2
[05:49:33+0000]   Using cached platformdirs-3.2.0-py3-none-any.whl (14 kB)
[05:49:33+0000] Collecting packaging>=22.0
[05:49:33+0000]   Using cached packaging-23.1-py3-none-any.whl (48 kB)
[05:49:33+0000] Collecting pathspec>=0.9.0
[05:49:33+0000]   Using cached pathspec-0.11.1-py3-none-any.whl (29 kB)
[05:49:34+0000] Collecting tomli>=1.1.0
[05:49:34+0000]   Using cached tomli-2.0.1-py3-none-any.whl (12 kB)
[05:49:34+0000] Collecting mypy-extensions>=0.4.3
[05:49:34+0000]   Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
[05:49:34+0000] Collecting click>=8.0.0
[05:49:34+0000]   Using cached click-8.1.3-py3-none-any.whl (96 kB)
[05:49:34+0000] Collecting dash-table==5.0.0
[05:49:34+0000]   Using cached dash_table-5.0.0-py3-none-any.whl (3.9 kB)
[05:49:34+0000] Collecting dash-html-components==2.0.0
[05:49:34+0000]   Using cached dash_html_components-2.0.0-py3-none-any.whl (4.1 kB)
[05:49:35+0000] Collecting dash-core-components==2.0.0
[05:49:35+0000]   Using cached dash_core_components-2.0.0-py3-none-any.whl (3.8 kB)
[05:49:35+0000] Collecting Flask>=1.0.4
[05:49:35+0000]   Using cached Flask-2.2.3-py3-none-any.whl (101 kB)
[05:49:35+0000] Collecting requests
[05:49:35+0000]   Using cached requests-2.28.2-py3-none-any.whl (62 kB)
[05:49:37+0000] Collecting numpy
[05:49:37+0000]   Using cached numpy-1.24.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
[05:49:37+0000] Collecting jinja2>=2.9
[05:49:38+0000]   Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
[05:49:38+0000] Collecting branca>=0.6.0
[05:49:38+0000]   Using cached branca-0.6.0-py3-none-any.whl (24 kB)
[05:49:38+0000] Collecting geographiclib<3,>=1.52
[05:49:38+0000]   Using cached geographiclib-2.0-py3-none-any.whl (40 kB)
[05:49:38+0000] Collecting typing-extensions>=3.10
[05:49:38+0000]   Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
[05:49:38+0000] Collecting tzdata>=2022.1
[05:49:38+0000]   Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
[05:49:39+0000] Collecting python-dateutil>=2.8.2
[05:49:39+0000]   Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
[05:49:39+0000] Collecting pytz>=2020.1
[05:49:39+0000]   Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
[05:49:39+0000] Collecting tenacity>=6.2.0
[05:49:40+0000]   Using cached tenacity-8.2.2-py3-none-any.whl (24 kB)
[05:49:41+0000] Collecting Werkzeug>=2.2.2
[05:49:41+0000]   Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
[05:49:42+0000] Collecting itsdangerous>=2.0
[05:49:42+0000]   Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
[05:49:42+0000] Collecting MarkupSafe>=2.0
[05:49:42+0000]   Using cached MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
[05:49:42+0000] Collecting six>=1.5
[05:49:42+0000]   Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
[05:49:43+0000] Collecting urllib3<1.27,>=1.21.1
[05:49:43+0000]   Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
[05:49:43+0000] Collecting certifi>=2017.4.17
[05:49:43+0000]   Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
[05:49:43+0000] Collecting charset-normalizer<4,>=2
[05:49:43+0000]   Using cached charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
[05:49:43+0000] Collecting idna<4,>=2.5
[05:49:43+0000]   Using cached idna-3.4-py3-none-any.whl (61 kB)
[05:49:45+0000] Installing collected packages: pytz, dash-table, dash-html-components, dash-core-components, urllib3, tzdata, typing-extensions, tomli, tenacity, six, python-dotenv, psycopg2, platformdirs, pathspec, packaging, numpy, mypy-extensions, MarkupSafe, itsdangerous, idna, gpxpy, geojson, geographiclib, click, charset-normalizer, certifi, Werkzeug, shapely, scipy, requests, rdp, python-dateutil, plotly, mypy, jinja2, geopy, black, pandas, Flask, branca, folium, dash
[05:51:31+0000] Successfully installed Flask-2.2.3 MarkupSafe-2.1.2 Werkzeug-2.2.3 black-23.3.0 branca-0.6.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 dash-2.9.3 dash-core-components-2.0.0 dash-html-components-2.0.0 dash-table-5.0.0 folium-0.14.0 geographiclib-2.0 geojson-3.0.1 geopy-2.3.0 gpxpy-1.5.0 idna-3.4 itsdangerous-2.1.2 jinja2-3.1.2 mypy-1.2.0 mypy-extensions-1.0.0 numpy-1.24.3 packaging-23.1 pandas-2.0.1 pathspec-0.11.1 platformdirs-3.2.0 plotly-5.14.1 psycopg2-2.9.6 python-dateutil-2.8.2 python-dotenv-1.0.0 pytz-2023.3 rdp-0.8 requests-2.28.2 scipy-1.10.1 shapely-2.0.1 six-1.16.0 tenacity-8.2.2 tomli-2.0.1 typing-extensions-4.5.0 tzdata-2023.3 urllib3-1.26.15

[notice] A new release of pip available: 22.2.2 -> 23.1.1
[notice] To update, run: pip install --upgrade pip
Not a vso image, so not writing build commands
Preparing output...

image image

But then it all crashes. The diagnostic resource provides logs:

2023-04-25T05:30:55.507802900Z    _____                               
2023-04-25T05:30:55.508221200Z   /  _  \ __________ _________   ____  
2023-04-25T05:30:55.508236800Z  /  /_\  \\___   /  |  \_  __ \_/ __ \ 
2023-04-25T05:30:55.508241600Z /    |    \/    /|  |  /|  | \/\  ___/ 
2023-04-25T05:30:55.508245600Z \____|__  /_____ \____/ |__|    \___  >
2023-04-25T05:30:55.508249900Z         \/      \/                  \/ 
2023-04-25T05:30:55.508254000Z A P P   S E R V I C E   O N   L I N U X
2023-04-25T05:30:55.508258100Z 
2023-04-25T05:30:55.508261900Z Documentation: http://aka.ms/webapp-linux
2023-04-25T05:30:55.508265700Z Python 3.10.10
2023-04-25T05:30:55.508269500Z Note: Any data outside '/home' is not persisted
2023-04-25T05:30:57.538205789Z Starting OpenBSD Secure Shell server: sshd.
2023-04-25T05:30:57.642961043Z Site's appCommandLine: src/app.py
2023-04-25T05:30:57.644116246Z Checking of src/app.py is a file
2023-04-25T05:30:57.649998560Z App command line is not a file on disk, using it as the startup command.
2023-04-25T05:30:57.992436589Z Starting periodic command scheduler: cron.
2023-04-25T05:30:58.000491608Z Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite -userStartupCommand 'src/app.py'
2023-04-25T05:30:58.164378105Z Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
2023-04-25T05:30:58.167577413Z Build Operation ID: 6d5949cb4ed73013
2023-04-25T05:30:58.168596015Z Oryx Version: 0.2.20230322.1, Commit: 6c04ec75685b441c9378c238b6491cf9812313db, ReleaseTagName: 20230322.1
2023-04-25T05:30:58.169264417Z Output is compressed. Extracting it...
2023-04-25T05:30:58.267656655Z Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8db454cd3b9c7e4'...
2023-04-25T05:31:25.232203281Z App path is set to '/tmp/8db454cd3b9c7e4'
2023-04-25T05:31:25.359397279Z Writing output script to '/opt/startup/startup.sh'
2023-04-25T05:31:27.054038252Z Using packages from virtual environment antenv located at /tmp/8db454cd3b9c7e4/antenv.
2023-04-25T05:31:27.055175755Z Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8db454cd3b9c7e4/antenv/lib/python3.10/site-packages'
2023-04-25T05:31:27.154097187Z src/app.py: 1: from: not found
2023-04-25T05:31:27.154120387Z src/app.py: 3: from: not found
2023-04-25T05:31:27.154126787Z src/app.py: 4: import: not found
2023-04-25T05:31:27.154131387Z src/app.py: 5: import: not found
2023-04-25T05:31:27.154135787Z src/app.py: 6: import: not found
2023-04-25T05:31:27.154140287Z src/app.py: 8: Syntax error: "(" unexpected

But this doesn't make any sense because the dependencies were clearly installed in the Oryx build logs. I can't SSH into the webapp to examine the file system because it fails to start up.

Any advice would be greatly appreciated.

pauld-msft commented 1 year ago

Hi @tjprescott, sorry you are having issues with your flask deployment.

Have you provided any custom configurations for the Oryx build or app service deployment? One thing that I've noticed is the -userStartupCommand 'src/app.py' flag in the logs you provided. Have you tried allowing app service to run the flask app with gunicorn, as documented in https://learn.microsoft.com/en-us/azure/app-service/configure-language-python#flask-app? Have you tried specifying python in the custom startup command, instead of just src/app.py?

You mentioned that you cannot SSH in the webapp for the file system. I assume that this prevents you from accessing the KUDU service as well? https://github.com/projectkudu/kudu/wiki/Accessing-the-kudu-service#overview

If there was some way for us to reproduce the issue, that would be immensely helpful. We have some sample flask apps (e.g. https://github.com/microsoft/Oryx/tree/main/tests/SampleApps/python/flask-app) that appear to be working.

tjprescott commented 1 year ago

As it turned out, the issue was that I had code like

app = Dash(___name__)

which needed to be converted to:

dash_app = Dash(___name__)
app = dash_app.server

Due to Azure WebApps looking very specifically for a Flask variable named "app".

pauld-msft commented 1 year ago

@tjprescott - Thanks for posting the fix here as well!