PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.88k stars 1.56k forks source link

Enhance documentation on Prefect Settings #10697

Open WillRaphaelson opened 1 year ago

WillRaphaelson commented 1 year ago

First check

Describe the issue

There is a lot of great advanced functionality enabled via settings, but many of them are not documented in a friendly way.

Describe the proposed change

Would be great to have a table of all the settings, what they do, and when one might use them. Even better would be something autodocumenting, that could parse these settings and generate the table / information we're looking for from descriptions.

Additional context

There are a lot of settings

PREFECT_PROFILE='prod' PREFECT_AGENT_PREFETCH_SECONDS='15' (from defaults) PREFECT_AGENT_QUERY_INTERVAL='15.0' (from defaults) PREFECT_API_BLOCKS_REGISTER_ON_START='True' (from defaults) PREFECT_API_DATABASE_CONNECTION_TIMEOUT='5.0' (from defaults) PREFECT_API_DATABASE_CONNECTION_URL='****' (from defaults) PREFECT_API_DATABASE_ECHO='False' (from defaults) PREFECT_API_DATABASE_MIGRATE_ON_START='True' (from defaults) PREFECT_API_DATABASE_PASSWORD='****' (from defaults) PREFECT_API_DATABASE_TIMEOUT='10.0' (from defaults) PREFECT_API_DEFAULT_LIMIT='200' (from defaults) PREFECT_API_ENABLE_HTTP2='True' (from defaults) PREFECT_API_KEY='****' (from profile) PREFECT_API_REQUEST_TIMEOUT='60.0' (from defaults) PREFECT_API_SERVICES_CANCELLATION_CLEANUP_ENABLED='True' (from defaults) PREFECT_API_SERVICES_CANCELLATION_CLEANUP_LOOP_SECONDS='20.0' (from defaults) PREFECT_API_SERVICES_FLOW_RUN_NOTIFICATIONS_ENABLED='True' (from defaults) PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS='0:00:05' (from defaults) PREFECT_API_SERVICES_LATE_RUNS_ENABLED='True' (from defaults) PREFECT_API_SERVICES_LATE_RUNS_LOOP_SECONDS='5.0' (from defaults) PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_ENABLED='True' (from defaults) PREFECT_API_SERVICES_PAUSE_EXPIRATIONS_LOOP_SECONDS='5.0' (from defaults) PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE='100' (from defaults) PREFECT_API_SERVICES_SCHEDULER_ENABLED='True' (from defaults) PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE='500' (from defaults) PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS='60.0' (from defaults) PREFECT_API_SERVICES_SCHEDULER_MAX_RUNS='100' (from defaults) PREFECT_API_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME='100 days, 0:00:00' (from defaults) PREFECT_API_SERVICES_SCHEDULER_MIN_RUNS='3' (from defaults) PREFECT_API_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME='1:00:00' (from defaults) PREFECT_API_TASK_CACHE_KEY_MAX_LENGTH='2000' (from defaults) PREFECT_API_TLS_INSECURE_SKIP_VERIFY='False' (from defaults) PREFECT_API_URL='https://api.prefect.cloud/api/accounts/9b649228-0419-40e1-9e0d- 44954b5c0ab6/workspaces/76e8a8a5-eaeb-4b97-9085-112edac86e1e' (from profile) PREFECT_ASYNC_FETCH_STATE_RESULT='False' (from defaults) PREFECT_CLIENT_MAX_RETRIES='5' (from defaults) PREFECT_CLIENT_RETRY_EXTRA_CODES='' (from defaults) PREFECT_CLIENT_RETRY_JITTER_FACTOR='0.2' (from defaults) PREFECT_CLI_COLORS='True' (from defaults) PREFECT_CLI_PROMPT='None' (from defaults) PREFECT_CLI_WRAP_LINES='True' (from defaults) PREFECT_CLOUD_API_URL='https://api.prefect.cloud/api' (from defaults) PREFECT_CLOUD_UI_URL='None' (from defaults) PREFECT_CLOUD_URL='None' (from defaults) PREFECT_DEBUG_MODE='False' (from defaults) PREFECT_EXPERIMENTAL_ENABLE_ARTIFACTS='True' (from defaults) PREFECT_EXPERIMENTAL_ENABLE_EVENTS_CLIENT='True' (from defaults) PREFECT_EXPERIMENTAL_ENABLE_WORKERS='True' (from defaults) PREFECT_EXPERIMENTAL_ENABLE_WORKSPACE_DASHBOARD='True' (from defaults) PREFECT_EXPERIMENTAL_ENABLE_WORK_POOLS='True' (from defaults) PREFECT_EXPERIMENTAL_WARN='True' (from defaults) PREFECT_EXPERIMENTAL_WARN_ARTIFACTS='False' (from defaults) PREFECT_EXPERIMENTAL_WARN_EVENTS_CLIENT='False' (from defaults) PREFECT_EXPERIMENTAL_WARN_VISUALIZE='True' (from defaults) PREFECT_EXPERIMENTAL_WARN_WORKERS='False' (from defaults) PREFECT_EXPERIMENTAL_WARN_WORKSPACE_DASHBOARD='False' (from defaults) PREFECT_EXPERIMENTAL_WARN_WORK_POOLS='False' (from defaults) PREFECT_EXTRA_ENTRYPOINTS='' (from defaults) PREFECT_FLOW_DEFAULT_RETRIES='0' (from defaults) PREFECT_FLOW_DEFAULT_RETRY_DELAY_SECONDS='0' (from defaults) PREFECT_HOME='~/.prefect' (from defaults) PREFECT_LOCAL_STORAGE_PATH='${PREFECT_HOME}/storage' (from defaults) PREFECT_LOGGING_COLORS='True' (from defaults) PREFECT_LOGGING_EXTRA_LOGGERS='' (from defaults) PREFECT_LOGGING_INTERNAL_LEVEL='ERROR' (from defaults) PREFECT_LOGGING_LEVEL='INFO' (from defaults) PREFECT_LOGGING_LOG_PRINTS='False' (from defaults) PREFECT_LOGGING_MARKUP='False' (from defaults) PREFECT_LOGGING_ORION_BATCH_INTERVAL='None' (from defaults) PREFECT_LOGGING_ORION_BATCH_SIZE='None' (from defaults) PREFECT_LOGGING_ORION_ENABLED='None' (from defaults) PREFECT_LOGGING_ORION_MAX_LOG_SIZE='None' (from defaults) PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW='None' (from defaults) PREFECT_LOGGING_SERVER_LEVEL='WARNING' (from defaults) PREFECT_LOGGING_SETTINGS_PATH='${PREFECT_HOME}/logging.yml' (from defaults) PREFECT_LOGGING_TO_API_BATCH_INTERVAL='2.0' (from defaults) PREFECT_LOGGING_TO_API_BATCH_SIZE='4000000' (from defaults) PREFECT_LOGGING_TO_API_ENABLED='True' (from defaults) PREFECT_LOGGING_TO_API_MAX_LOG_SIZE='1000000' (from defaults) PREFECT_LOGGING_TO_API_WHEN_MISSING_FLOW='warn' (from defaults) PREFECT_MEMOIZE_BLOCK_AUTO_REGISTRATION='True' (from defaults) PREFECT_MEMO_STORE_PATH='${PREFECT_HOME}/memo_store.toml' (from defaults) PREFECT_ORION_ANALYTICS_ENABLED='None' (from defaults) PREFECT_ORION_API_DEFAULT_LIMIT='None' (from defaults) PREFECT_ORION_API_HOST='None' (from defaults) PREFECT_ORION_API_KEEPALIVE_TIMEOUT='None' (from defaults) PREFECT_ORION_API_PORT='None' (from defaults) PREFECT_ORION_BLOCKS_REGISTER_ON_START='None' (from defaults) PREFECT_ORION_DATABASE_CONNECTION_TIMEOUT='None' (from defaults) PREFECT_ORION_DATABASE_CONNECTION_URL='None' (from defaults) PREFECT_ORION_DATABASE_ECHO='None' (from defaults) PREFECT_ORION_DATABASE_MIGRATE_ON_START='None' (from defaults) PREFECT_ORION_DATABASE_PASSWORD='None' (from defaults) PREFECT_ORION_DATABASE_TIMEOUT='None' (from defaults) PREFECT_ORION_SERVICES_CANCELLATION_CLEANUP_ENABLED='None' (from defaults) PREFECT_ORION_SERVICES_CANCELLATION_CLEANUP_LOOP_SECONDS='None' (from defaults) PREFECT_ORION_SERVICES_FLOW_RUN_NOTIFICATIONS_ENABLED='None' (from defaults) PREFECT_ORION_SERVICES_LATE_RUNS_AFTER_SECONDS='None' (from defaults) PREFECT_ORION_SERVICES_LATE_RUNS_ENABLED='None' (from defaults) PREFECT_ORION_SERVICES_LATE_RUNS_LOOP_SECONDS='None' (from defaults) PREFECT_ORION_SERVICES_PAUSE_EXPIRATIONS_ENABLED='None' (from defaults) PREFECT_ORION_SERVICES_PAUSE_EXPIRATIONS_LOOP_SECONDS='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_ENABLED='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_INSERT_BATCH_SIZE='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_LOOP_SECONDS='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_MAX_RUNS='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_MIN_RUNS='None' (from defaults) PREFECT_ORION_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME='None' (from defaults) PREFECT_ORION_TASK_CACHE_KEY_MAX_LENGTH='None' (from defaults) PREFECT_ORION_UI_API_URL='None' (from defaults) PREFECT_ORION_UI_ENABLED='None' (from defaults) PREFECT_PROFILES_PATH='${PREFECT_HOME}/profiles.toml' (from defaults) PREFECT_RESULTS_DEFAULT_SERIALIZER='pickle' (from defaults) PREFECT_RESULTS_PERSIST_BY_DEFAULT='False' (from defaults) PREFECT_RUNNER_PROCESS_LIMIT='5' (from defaults) PREFECT_SERVER_ANALYTICS_ENABLED='True' (from defaults) PREFECT_SERVER_API_HOST='127.0.0.1' (from defaults) PREFECT_SERVER_API_KEEPALIVE_TIMEOUT='5' (from defaults) PREFECT_SERVER_API_PORT='4200' (from defaults) PREFECT_SQLALCHEMY_MAX_OVERFLOW='None' (from defaults) PREFECT_SQLALCHEMY_POOL_SIZE='None' (from defaults) PREFECT_TASKS_REFRESH_CACHE='False' (from defaults) PREFECT_TASK_DEFAULT_RETRIES='0' (from defaults) PREFECT_TASK_DEFAULT_RETRY_DELAY_SECONDS='0.0' (from defaults) PREFECT_TEST_MODE='False' (from defaults) PREFECT_TEST_SETTING='None' (from defaults) PREFECT_UI_API_URL='None' (from defaults) PREFECT_UI_ENABLED='True' (from defaults) PREFECT_UI_URL='None' (from defaults) PREFECT_UNIT_TEST_MODE='False' (from defaults) PREFECT_WORKER_HEARTBEAT_SECONDS='30.0' (from defaults) PREFECT_WORKER_PREFETCH_SECONDS='10.0' (from defaults) PREFECT_WORKER_QUERY_SECONDS='10.0' (from defaults)

serinamarie commented 1 year ago

Some setting adjustments are even known workarounds for issues, so that might be nice to call out as well.