superset error when testing

apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform

Apache License 2.0

62.04k stars 13.61k forks source link

Bug description

with our team we did load testing on superset dashboards we created. used selenium, pytest to implement the tests and run 10 workers parallel using the pystest-xdist library. 6 of the workers did fine but I got different errors on 4 workers . one of the errors: {"errors": [{"message": "429 Too Many Requests: 50 per 1 second", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - Superset encountered an unexpected error."}]}}]}, and other errors are: SupersetAPIError, ChunkLoadError, DBEngineError. and screenshots of all errors are attached below. what is the problem causing these errors? is it because of superset settings, or configurations we have used or is there any other cause? should I specify my configurations to find the reason of error to find solution? any help is appreciated.

How to reproduce the bug

Снимок4 Снимок2 Снимок1 Снимок3

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

I don't know

Node version

Browser

Chrome

Additional context

No response

Checklist

[ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
[ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

The errors you're encountering during load testing on Superset dashboards are likely due to a combination of rate limiting and database connection pool limits. Here are some steps to address these issues:

Increase Database Connection Pool Size: Adjust the database connection pool settings to allow more concurrent connections. You can do this by modifying the SQLALCHEMY_DATABASE_URI and SQLALCHEMY_ENGINE_OPTIONS in your superset_config.py file:

# superset_config.py

SQLALCHEMY_DATABASE_URI = (
   'postgresql+psycopg2://user:password@localhost/mydatabase'
   '?pool_size=20&max_overflow=10&pool_timeout=30&pool_recycle=3600'
)

SQLALCHEMY_ENGINE_OPTIONS = {
   'pool_size': 20,
   'max_overflow': 10,
   'pool_timeout': 30,
   'pool_recycle': 3600,
}

Customize Rate Limiting Settings: Increase the rate limits to handle higher loads by modifying the rate limiting settings in your superset_config.py file:
```
RATELIMIT_APPLICATION = "100 per second"
AUTH_RATE_LIMIT = "10 per second"
RATELIMIT_STORAGE_URI = "redis://localhost:6379"
```

Implement Caching: Use caching mechanisms to reduce the number of direct database queries. Configure caching in your superset_config.py file:

CACHE_CONFIG = {
   'CACHE_TYPE': 'redis',
   'CACHE_DEFAULT_TIMEOUT': 300,
   'CACHE_KEY_PREFIX': 'superset_',
   'CACHE_REDIS_HOST': 'localhost',
   'CACHE_REDIS_PORT': 6379,
   'CACHE_REDIS_DB': 1,
   'CACHE_REDIS_URL': 'redis://localhost:6379/1'
}

DATA_CACHE_CONFIG = CACHE_CONFIG

Rate Limiting in Load Testing Scripts: Implement rate limiting in your Selenium and pytest scripts to avoid overwhelming the server with too many concurrent requests.

By following these steps, you should be able to mitigate the errors related to rate limiting and database connection pool limits during your load testing ^[1]^[2]^[3].

To continue talking to Dosu, mention @dosu.

apache / superset