Internal Server Error - on Google Cloud Deployment

[Neitsup]

Hi,

thanks for the great project! I have successfully managed to run the code on my machine. Everything works as desired.

Now I wanted to run the script via the Google Cloud. I followed the instructions, nevertheless I'm not able to get it running. I don't get any telegram messages and when I open the url to view the application run in the web browser (https://flathunterXXX.ew.r.appspot.com), it only shows me:

Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Do you know where the error lies?

This is the process to deploy the app: ############################################ START ############################################

C:\Users\USER\PycharmProjects\flathunter>gcloud app deploy Services to deploy:

descriptor: [C:\Users\USER\PycharmProjects\flathunter\app.yaml] source: [C:\Users\USER\PycharmProjects\flathunter] target project: [flathunterXXXX] target service: [default] target version: [20220413t18XXXX] target url: [https://flathunterXXX.ew.r.appspot.com] target service account: [App Engine default service account]

Do you want to continue (Y/n)? Y

Beginning deployment of service [default]...

============================================================

= Uploading 1 file to Google Cloud Storage =

============================================================

File upload done. Updating service [default]...done. Setting traffic split for service [default]...done. Deployed service [default] to [https://flathunterXXX.ew.r.appspot.com]

You can stream logs from the command line by running: $ gcloud app logs tail -s default

To view your application in the web browser run: $ gcloud app browse

C:\Users\USER\PycharmProjects\flathunter>gcloud app deploy cron.yaml Configurations to update:

descriptor: [C:\Users\USER\PycharmProjects\flathunter\cron.yaml] type: [cron jobs] target project: [flathunterXXXX]

Do you want to continue (Y/n)? Y

Updating config [cron]...-WARNING: We are using the App Engine app location (europe-west1) as the default location. Please use the "--location" flag if you want to use a different location. Updating config [cron]...|WARNING: We are using the App Engine app location (europe-west1) as the default location. Please use the "--location" flag if you want to use a different location. Updating config [cron]...done.

Cron jobs have been updated.

Visit the Cloud Platform Console Task Queues page to view your queues and cron jobs. https://console.cloud.google.com/appengine/taskqueues/cron?project=flathunterXXXX

############################################# END #############################################

And this is my config.yaml file: config.txt

[codders]

Hi there,

The clues would for sure be in the Google cloud logs. I would guess an out of date requirements.txt, but I don't think many people are using the Google cloud deployment so you might have to do some digging.

Neitsup @.***> schrieb am Mi., 13. Apr. 2022, 18:38:

Hi,

thanks for the great project! I have successfully managed to run the code on my machine. Everything works as desired.

Now I wanted to run the script via the Google Cloud. I followed the instructions, nevertheless I'm not able to get it running. I don't get any telegram messages and when I open the url to view the application run in the web browser (https://flathunterXXX.ew.r.appspot.com https://flathunterXXX.ew.r.appspot.com), it only shows me:

Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Do you know where the error lies?

This is the process to deploy the app: ############################################ START ############################################

C:\Users\USER\PycharmProjects\flathunter>gcloud app deploy Services to deploy:

descriptor: [C:\Users\USER\PycharmProjects\flathunter\app.yaml] source: [C:\Users\USER\PycharmProjects\flathunter] target project: [flathunterXXXX] target service: [default] target version: [20220413t18XXXX] target url: [https://flathunterXXX.ew.r.appspot.com] target service account: [App Engine default service account]

Do you want to continue (Y/n)? Y

Beginning deployment of service [default]...

============================================================

= Uploading 1 file to Google Cloud Storage =

============================================================

File upload done. Updating service [default]...done. Setting traffic split for service [default]...done. Deployed service [default] to [https://flathunterXXX.ew.r.appspot.com]

You can stream logs from the command line by running: $ gcloud app logs tail -s default

To view your application in the web browser run: $ gcloud app browse

C:\Users\USER\PycharmProjects\flathunter>gcloud app deploy cron.yaml Configurations to update:

descriptor: [C:\Users\USER\PycharmProjects\flathunter\cron.yaml] type: [cron jobs] target project: [flathunterXXXX]

Do you want to continue (Y/n)? Y

Updating config [cron]...-WARNING: We are using the App Engine app location (europe-west1) as the default location. Please use the "--location" flag if you want to use a different location. Updating config [cron]...|WARNING: We are using the App Engine app location (europe-west1) as the default location. Please use the "--location" flag if you want to use a different location. Updating config [cron]...done.

Cron jobs have been updated.

Visit the Cloud Platform Console Task Queues page to view your queues and cron jobs.

https://console.cloud.google.com/appengine/taskqueues/cron?project=flathunterXXXX

############################################# END #############################################

And this is my config.yaml file: config.txt https://github.com/flathunters/flathunter/files/8483014/config.txt

— Reply to this email directly, view it on GitHub https://github.com/flathunters/flathunter/issues/167, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEK5SMYVBQCA6UDUSVNELVE32AZANCNFSM5TLHWKGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[abuchmueller]

Try pip freeze > requirements.txt in the flathunter directory, then redeploy to google by running gcloud app deploy. Then deploy cron an configure Firestone and cloud build like in the Readme. And then you need to configure the login widget, which I am currently struggling with because the Readme offers no information on that topic and in the YAML only references https://core.telegram.org/widgets/login without explaining how to set the parameters in the config.yaml, like where for example do I get the session_key for the login widget?

[codders]

@abuchmueller The session key just has to be a secret value that is private for your deploy - it can be anything. You can use something like pwgen -n 20 -y -s to generate it. The only other thing you need is the bot token, which hopefully the Telegram docs explain. But if you have more questions please open a separate issue about it - would be great to update the README based on the problems you've bumped into.

[abuchmueller]

@codders Thanks for answering. I had put something non-sensical like "asdf" in it and it seemed to work. I still don't get notifications from it, though, only if I run flathunt.py locally it works. I will write a new issue on this later this evening, and gladly do a PR and update the Readme if we can get it to work properly.

[codders]

I don't know if you can look in the logs for the cloud deployment to see if it's crawling okay. There is something with Google cloud where you need to be in a paid tier (i.e. have your credit card saved there) to get external network access for your deployments - maybe worth checking that.

[abuchmueller]

I tried gcloud app logs read and get the following.

2022-08-22 10:05:02 default[20220822t120111]  "GET /login_with_telegram?id=XXXXXX&first_name=Andreas&auth_date=1661162702&hash=XXXX HTTP/1.1" 302
2022-08-22 10:05:02 default[20220822t120111]  [2022-08-22 10:05:02 +0000] [11] [INFO] Starting gunicorn 20.1.0
2022-08-22 10:05:02 default[20220822t120111]  [2022-08-22 10:05:02 +0000] [11] [INFO] Listening at: http://0.0.0.0:8081 (11)
2022-08-22 10:05:02 default[20220822t120111]  [2022-08-22 10:05:02 +0000] [11] [INFO] Using worker: gthread
2022-08-22 10:05:02 default[20220822t120111]  [2022-08-22 10:05:02 +0000] [20] [INFO] Booting worker with pid: 20
2022-08-22 10:05:02 default[20220822t120111]  [2022-08-22 10:05:02 +0000] [23] [INFO] Booting worker with pid: 23
2022-08-22 10:05:07 default[20220822t120111]  [2022/08/22 10:05:07|config.py               |INFO    ]: Using config /srv/flathunter/../config.yaml
2022-08-22 10:05:07 default[20220822t120111]  [2022/08/22 10:05:07|config.py               |INFO    ]: Using config /srv/flathunter/../config.yaml
2022-08-22 10:05:07 default[20220822t120111]  [2022/08/22 10:05:07|config.py               |INFO    ]: Using config /srv/flathunter/../config.yaml
2022-08-22 10:05:07 default[20220822t120111]  [2022/08/22 10:05:07|config.py               |INFO    ]: Using config /srv/flathunter/../config.yaml
2022-08-22 10:05:07 default[20220822t120111]  [2022/08/22 10:05:07|views.py                |INFO    ]: User is: {'id': 'XXXXXXXX', 'first_name': 'Andreas', 'auth_date': '1661162702'}
2022-08-22 10:05:07 default[20220822t120111]  "GET / HTTP/1.1" 200
2022-08-22 10:05:07 default[20220822t120111]  "GET /static/style.css HTTP/1.1" 304
2022-08-22 10:05:07 default[20220822t120111]  "GET /static/GitHub-Mark-32px.png HTTP/1.1" 304
2022-08-22 10:05:07 default[20220822t120111]  "GET /static/site.webmanifest HTTP/1.1" 304
2022-08-22 10:10:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 10:13:46 default[20220822t114648]  [2022-08-22 10:13:46 +0000] [10] [INFO] Handling signal: term
2022-08-22 10:13:46 default[20220822t114648]  [2022-08-22 10:13:46 +0000] [21] [INFO] Worker exiting (pid: 21)
2022-08-22 10:13:47 default[20220822t114648]  [2022-08-22 10:13:47 +0000] [24] [INFO] Worker exiting (pid: 24)
2022-08-22 10:13:47 default[20220822t114648]  [2022-08-22 10:13:47 +0000] [10] [INFO] Shutting down: Master
2022-08-22 10:15:03 default[20220822t114648]  [2022-08-22 10:15:03 +0000] [10] [INFO] Handling signal: term
2022-08-22 10:15:03 default[20220822t114648]  [2022-08-22 10:15:03 +0000] [21] [INFO] Worker exiting (pid: 21)
2022-08-22 10:15:03 default[20220822t114648]  [2022-08-22 10:15:03 +0000] [24] [INFO] Worker exiting (pid: 24)
2022-08-22 10:15:05 default[20220822t114648]  [2022-08-22 10:15:05 +0000] [10] [INFO] Shutting down: Master
2022-08-22 10:20:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 10:21:09 default[20220822t120111]  [2022-08-22 10:21:09 +0000] [11] [INFO] Handling signal: term
2022-08-22 10:21:09 default[20220822t120111]  [2022-08-22 10:21:09 +0000] [23] [INFO] Worker exiting (pid: 23)
2022-08-22 10:21:09 default[20220822t120111]  [2022-08-22 10:21:09 +0000] [20] [INFO] Worker exiting (pid: 20)
2022-08-22 10:21:10 default[20220822t120111]  [2022-08-22 10:21:10 +0000] [11] [INFO] Shutting down: Master
2022-08-22 10:30:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 10:40:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 10:50:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 11:00:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 11:10:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 11:20:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 11:30:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201
2022-08-22 11:40:00 default[20220822t120111]  "GET /hunt HTTP/1.1" 201

Billing is enabled and a cc is linked. Basically, the bot calls /hunt gets {"body":"<div class=\"exposes row\">\n \n</div>","completedAt":"2022-08-22 12:16:32.092169+00:00","status":"Success"} returned and then hibernates for 10 minutes again. I'll write up a more concise issue of what I tried and what's not working currently later but so far Docker doesn't work (Intel and M1) and GCP also doesn't work - the only functioning way is to run this app is locally.

[codders]

Looks like the log level is Info. Can you deploy with Debug?

[abuchmueller]

This is the head of my deployed yaml config.

# Enable verbose mode (print DEBUG log messages)
verbose: True

Locally the debugger logs at the DEBUG level with this config, but on GCloud the above is all I can get with gcloud app logs read.

[codders]

I've updated the code in my latest pr (#204) so that debug logging works. Can you redeploy with the latest and see if there's more info there about the error?

[abuchmueller]

I've updated the code in my latest pr (#204) so that debug logging works. Can you redeploy with the latest and see if there's more info there about the error?

Thanks! I'll try it out later and report back.

[codders]

@abuchmueller any update here? If not, I would close this ticket.

[abuchmueller]

@abuchmueller any update here? If not, I would close this ticket.

Honestly, after you've patched gcloud run functionality in, I've forgotten completely about GAE. Feel free to close.