Fix remaining D401 checks

Taragolis commented 7 months ago

Body

Part of https://github.com/apache/airflow/issues/10742. Recently we enable D401 pydocstyle rule, however we still have some modules which do not follow D401 rule yet

Easy way to find all linting problem into the module

Remove corresponding line to particular module into the pyproject.toml in [tool.ruff.lint.per-file-ignores]
Run ruff via pre-commit hook

❯ pre-commit run ruff --all-files
Run 'ruff' for extremely fast Python linting.............................Failed
- hook id: ruff
- exit code: 1

airflow/www/auth.py:90:5: D401 First line of docstring should be in imperative mood: "This decorator is used to check permissions on views."
airflow/www/auth.py:348:5: D401 First line of docstring should be in imperative mood: "Decorator that checks current user's permissions to access the website."
Found 2 errors.

Airflow Core

[x] airflow/auth/managers/utils/fab.py
[x] airflow/decorators/bash.py
[x] airflow/executors/debug_executor.py
[x] airflow/models/baseoperator.py
[x] airflow/models/dagrun.py
[x] airflow/models/taskinstance.py
[x] airflow/operators/python.py
[x] airflow/plugins_manager.py
[x] airflow/providers_manager.py
[x] airflow/serialization/serde.py
[x] airflow/utils/log/task_context_logger.py
[x] airflow/utils/sqlalchemy.py
[x] airflow/www/auth.py
[x] airflow/www/blueprints.py
[x] airflow/www/views.py
[x] airflow/utils/file.py

Provider airbyte

[x] airflow/providers/airbyte/hooks/airbyte.py
[x] airflow/providers/airbyte/operators/airbyte.py
[x] airflow/providers/airbyte/sensors/airbyte.py
[x] airflow/providers/airbyte/triggers/airbyte.py

Provider amazon

[x] airflow/providers/amazon/aws/auth_manager/cli/avp_commands.py
[x] airflow/providers/amazon/aws/auth_manager/views/auth.py
[x] airflow/providers/amazon/aws/executors/ecs/boto_schema.py
[x] airflow/providers/amazon/aws/executors/ecs/ecs_executor.py
[x] airflow/providers/amazon/aws/executors/ecs/utils.py
[x] airflow/providers/amazon/aws/hooks/redshift_sql.py
[x] airflow/providers/amazon/aws/hooks/logs.py
[x] airflow/providers/amazon/aws/triggers/redshift_cluster.py
[x] airflow/providers/amazon/aws/triggers/redshift_data.py
[x] airflow/providers/amazon/aws/triggers/sagemaker.py

Provider apache.hive

[x] airflow/providers/apache/hive/hooks/hive.py

Provider apache.spark

[x] airflow/providers/apache/spark/hooks/spark_connect.py
[x] airflow/providers/apache/spark/hooks/spark_sql.py
[x] airflow/providers/apache/spark/hooks/spark_submit.py

Provider atlassian.jira

[x] airflow/providers/atlassian/jira/hooks/jira.py

Provider cncf.kubernetes

[x] airflow/providers/cncf/kubernetes/callbacks.py
[x] airflow/providers/cncf/kubernetes/operators/custom_object_launcher.py
[x] airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
[x] airflow/providers/cncf/kubernetes/resource_convert/configmap.py
[x] airflow/providers/cncf/kubernetes/resource_convert/env_variable.py
[x] airflow/providers/cncf/kubernetes/resource_convert/secret.py

Provider common.io

[x] airflow/providers/common/io/xcom/backend.py

Provider databricks

[x] airflow/providers/databricks/hooks/databricks.py
[x] airflow/providers/databricks/operators/databricks.py

Provider discord

[x] airflow/providers/discord/hooks/discord_webhook.py

Provider fab

[x] airflow/providers/fab/auth_manager/decorators/auth.py
[x] airflow/providers/fab/auth_manager/fab_auth_manager.py
[x] airflow/providers/fab/auth_manager/security_manager/override.py

Provider google

[x] airflow/providers/google/cloud/hooks/automl.py
[x] airflow/providers/google/cloud/hooks/bigquery.py
[x] airflow/providers/google/cloud/hooks/bigquery_dts.py
[x] airflow/providers/google/cloud/hooks/bigtable.py
[x] airflow/providers/google/cloud/hooks/cloud_batch.py
[x] airflow/providers/google/cloud/hooks/cloud_build.py
[x] airflow/providers/google/cloud/hooks/cloud_composer.py
[x] airflow/providers/google/cloud/hooks/cloud_memorystore.py
[x] airflow/providers/google/cloud/hooks/cloud_run.py
[x] airflow/providers/google/cloud/hooks/cloud_sql.py
[x] airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
[x] airflow/providers/google/cloud/hooks/compute.py
[x] airflow/providers/google/cloud/hooks/datacatalog.py
[x] airflow/providers/google/cloud/hooks/dataflow.py
[x] airflow/providers/google/cloud/hooks/dataform.py
[x] airflow/providers/google/cloud/hooks/datafusion.py
[x] airflow/providers/google/cloud/hooks/datapipeline.py
[x] airflow/providers/google/cloud/hooks/dataplex.py
[x] airflow/providers/google/cloud/hooks/dataprep.py
[x] airflow/providers/google/cloud/hooks/dataproc_metastore.py
[x] airflow/providers/google/cloud/hooks/datastore.py
[x] airflow/providers/google/cloud/hooks/dlp.py
[x] airflow/providers/google/cloud/hooks/gcs.py
[x] airflow/providers/google/cloud/hooks/gdm.py
[x] airflow/providers/google/cloud/hooks/kms.py
[x] airflow/providers/google/cloud/hooks/kubernetes_engine.py
[x] airflow/providers/google/cloud/hooks/life_sciences.py
[x] airflow/providers/google/cloud/hooks/looker.py
[x] airflow/providers/google/cloud/hooks/mlengine.py
[x] airflow/providers/google/cloud/hooks/natural_language.py
[x] airflow/providers/google/cloud/hooks/os_login.py
[x] airflow/providers/google/cloud/hooks/pubsub.py
[x] airflow/providers/google/cloud/hooks/secret_manager.py
[x] airflow/providers/google/cloud/hooks/spanner.py
[x] airflow/providers/google/cloud/hooks/speech_to_text.py
[x] airflow/providers/google/cloud/hooks/stackdriver.py
[x] airflow/providers/google/cloud/hooks/tasks.py
[x] airflow/providers/google/cloud/hooks/text_to_speech.py
[x] airflow/providers/google/cloud/hooks/translate.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/auto_ml.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/batch_prediction_job.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/custom_job.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/dataset.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/endpoint_service.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/hyperparameter_tuning_job.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/model_service.py
[x] airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py
[x] airflow/providers/google/cloud/hooks/video_intelligence.py
[x] airflow/providers/google/cloud/hooks/vision.py
[x] airflow/providers/google/cloud/hooks/workflows.py
[x] airflow/providers/google/cloud/links/vertex_ai.py
[x] airflow/providers/google/cloud/log/stackdriver_task_handler.py
[x] airflow/providers/google/cloud/operators/bigquery.py
[x] airflow/providers/google/cloud/operators/bigquery_dts.py
[x] airflow/providers/google/cloud/operators/cloud_base.py
[x] airflow/providers/google/cloud/operators/cloud_build.py
[x] airflow/providers/google/cloud/operators/cloud_sql.py
[x] airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py
[x] airflow/providers/google/cloud/operators/dataflow.py
[x] airflow/providers/google/cloud/operators/datafusion.py
[x] airflow/providers/google/cloud/operators/dataplex.py
[x] airflow/providers/google/cloud/operators/dataproc.py
[x] airflow/providers/google/cloud/operators/functions.py
[x] airflow/providers/google/cloud/operators/gcs.py
[x] airflow/providers/google/cloud/operators/kubernetes_engine.py
[x] airflow/providers/google/cloud/operators/mlengine.py
[x] airflow/providers/google/cloud/operators/pubsub.py
[x] airflow/providers/google/cloud/operators/spanner.py
[x] airflow/providers/google/cloud/operators/vertex_ai/auto_ml.py
[x] airflow/providers/google/cloud/operators/vertex_ai/batch_prediction_job.py
[x] airflow/providers/google/cloud/operators/vertex_ai/custom_job.py
[x] airflow/providers/google/cloud/operators/vertex_ai/hyperparameter_tuning_job.py
[x] airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py
[x] airflow/providers/google/cloud/sensors/bigquery.py
[x] airflow/providers/google/cloud/sensors/cloud_composer.py
[x] airflow/providers/google/cloud/sensors/gcs.py
[x] airflow/providers/google/cloud/sensors/pubsub.py
[x] airflow/providers/google/cloud/transfers/bigquery_to_gcs.py
[x] airflow/providers/google/cloud/transfers/bigquery_to_sql.py
[x] airflow/providers/google/cloud/transfers/cassandra_to_gcs.py
[x] airflow/providers/google/cloud/transfers/gcs_to_bigquery.py
[x] airflow/providers/google/cloud/transfers/gcs_to_gcs.py
[x] airflow/providers/google/cloud/transfers/gcs_to_sftp.py
[x] airflow/providers/google/cloud/transfers/local_to_gcs.py
[x] airflow/providers/google/cloud/transfers/mssql_to_gcs.py
[x] airflow/providers/google/cloud/transfers/mysql_to_gcs.py
[x] airflow/providers/google/cloud/transfers/oracle_to_gcs.py
[x] airflow/providers/google/cloud/transfers/postgres_to_gcs.py
[x] airflow/providers/google/cloud/transfers/presto_to_gcs.py
[x] airflow/providers/google/cloud/transfers/s3_to_gcs.py
[x] airflow/providers/google/cloud/transfers/sftp_to_gcs.py
[x] airflow/providers/google/cloud/transfers/sql_to_gcs.py
[x] airflow/providers/google/cloud/transfers/trino_to_gcs.py
[x] airflow/providers/google/cloud/triggers/bigquery.py
[x] airflow/providers/google/cloud/triggers/bigquery_dts.py
[x] airflow/providers/google/cloud/triggers/cloud_batch.py
[x] airflow/providers/google/cloud/triggers/cloud_build.py
[x] airflow/providers/google/cloud/triggers/cloud_run.py
[x] airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
[x] airflow/providers/google/cloud/triggers/dataflow.py
[x] airflow/providers/google/cloud/triggers/datafusion.py
[x] airflow/providers/google/cloud/triggers/dataplex.py
[x] airflow/providers/google/cloud/triggers/dataproc.py
[x] airflow/providers/google/cloud/triggers/gcs.py
[x] airflow/providers/google/cloud/triggers/kubernetes_engine.py
[x] airflow/providers/google/cloud/triggers/mlengine.py
[x] airflow/providers/google/cloud/triggers/pubsub.py
[x] airflow/providers/google/cloud/utils/bigquery.py
[x] airflow/providers/google/cloud/utils/credentials_provider.py
[x] airflow/providers/google/cloud/utils/dataform.py
[x] airflow/providers/google/cloud/utils/field_validator.py
[x] airflow/providers/google/cloud/utils/helpers.py
[x] airflow/providers/google/cloud/utils/mlengine_operator_utils.py
[x] airflow/providers/google/cloud/utils/mlengine_prediction_summary.py
[x] airflow/providers/google/common/auth_backend/google_openid.py
[x] airflow/providers/google/common/hooks/base_google.py
[x] airflow/providers/google/common/hooks/discovery_api.py
[x] airflow/providers/google/common/utils/id_token_credentials.py
[x] airflow/providers/google/firebase/hooks/firestore.py
[x] airflow/providers/google/leveldb/hooks/leveldb.py
[x] airflow/providers/google/marketing_platform/hooks/analytics.py
[x] airflow/providers/google/marketing_platform/hooks/campaign_manager.py
[x] airflow/providers/google/marketing_platform/hooks/display_video.py
[x] airflow/providers/google/marketing_platform/hooks/search_ads.py
[x] airflow/providers/google/suite/hooks/calendar.py
[x] airflow/providers/google/suite/hooks/drive.py
[x] airflow/providers/google/suite/hooks/sheets.py

Provider grpc

[x] airflow/providers/grpc/hooks/grpc.py

Provider hashicorp

[x] airflow/providers/hashicorp/_internal_client/vault_client.py
[x] airflow/providers/hashicorp/hooks/vault.py

Provider http

[x] airflow/providers/http/operators/http.py
[x] airflow/providers/http/triggers/http.py

Provider imap

[ ] airflow/providers/imap/hooks/imap.py

Provider jenkins

[x] airflow/providers/jenkins/hooks/jenkins.py

Provider microsoft.azure

[x] airflow/providers/microsoft/azure/hooks/adx.py
[x] airflow/providers/microsoft/azure/hooks/asb.py
[x] airflow/providers/microsoft/azure/hooks/base_azure.py
[x] airflow/providers/microsoft/azure/hooks/batch.py
[x] airflow/providers/microsoft/azure/hooks/container_instance.py
[x] airflow/providers/microsoft/azure/hooks/container_registry.py
[x] airflow/providers/microsoft/azure/hooks/container_volume.py
[x] airflow/providers/microsoft/azure/hooks/cosmos.py
[x] airflow/providers/microsoft/azure/hooks/data_factory.py
[x] airflow/providers/microsoft/azure/hooks/data_lake.py
[x] airflow/providers/microsoft/azure/hooks/fileshare.py
[x] airflow/providers/microsoft/azure/hooks/synapse.py
[x] airflow/providers/microsoft/azure/hooks/wasb.py
[x] airflow/providers/microsoft/azure/log/wasb_task_handler.py
[x] airflow/providers/microsoft/azure/operators/adx.py
[x] airflow/providers/microsoft/azure/operators/asb.py
[x] airflow/providers/microsoft/azure/operators/data_factory.py
[x] airflow/providers/microsoft/azure/operators/synapse.py
[x] airflow/providers/microsoft/azure/sensors/data_factory.py
[x] airflow/providers/microsoft/azure/sensors/wasb.py
[x] airflow/providers/microsoft/azure/transfers/sftp_to_wasb.py
[x] airflow/providers/microsoft/azure/triggers/data_factory.py
[x] airflow/providers/microsoft/azure/triggers/wasb.py

Provider microsoft.mssql

[x] airflow/providers/microsoft/mssql/hooks/mssql.py

Provider microsoft.psrp

[x] airflow/providers/microsoft/psrp/hooks/psrp.py

Provider mongo

[x] airflow/providers/mongo/hooks/mongo.py

Provider mysql

[x] airflow/providers/mysql/hooks/mysql.py
[x] airflow/providers/mysql/transfers/s3_to_mysql.py

Provider neo4j

[x] airflow/providers/neo4j/hooks/neo4j.py

Provider odbc

[x] airflow/providers/odbc/hooks/odbc.py

Provider openfaas

[x] airflow/providers/openfaas/hooks/openfaas.py

Provider openlineage

[x] airflow/providers/openlineage/plugins/adapter.py
[x] airflow/providers/openlineage/sqlparser.py
[x] airflow/providers/openlineage/utils/sql.py

Provider opensearch

[x] airflow/providers/opensearch/hooks/opensearch.py
[x] airflow/providers/opensearch/operators/opensearch.py

Provider opsgenie

[x] airflow/providers/opsgenie/hooks/opsgenie.py

Provider oracle

[x] airflow/providers/oracle/hooks/oracle.py

Provider pagerduty

[x] airflow/providers/pagerduty/hooks/pagerduty.py
[x] airflow/providers/pagerduty/hooks/pagerduty_events.py

Provider papermill

[x] airflow/providers/papermill/hooks/kernel.py

Provider pinecone

[x] airflow/providers/pinecone/hooks/pinecone.py

Provider postgres

[x] airflow/providers/postgres/hooks/postgres.py

Provider presto

[x] airflow/providers/presto/hooks/presto.py

Provider qdrant

[x] airflow/providers/qdrant/hooks/qdrant.py

Provider redis

[x] airflow/providers/redis/hooks/redis.py

Provider salesforce

[x] airflow/providers/salesforce/hooks/salesforce.py
[x] airflow/providers/salesforce/operators/bulk.py
[x] airflow/providers/salesforce/operators/salesforce_apex_rest.py

Provider samba

[x] airflow/providers/samba/hooks/samba.py
[x] airflow/providers/samba/transfers/gcs_to_samba.py

Provider segment

[x] airflow/providers/segment/hooks/segment.py

Provider sftp

[x] airflow/providers/sftp/decorators/sensors/sftp.py
[x] airflow/providers/sftp/hooks/sftp.py
[x] airflow/providers/sftp/operators/sftp.py
[x] airflow/providers/sftp/sensors/sftp.py
[x] airflow/providers/sftp/triggers/sftp.py

Provider slack

[x] airflow/providers/slack/hooks/slack.py
[x] airflow/providers/slack/hooks/slack_webhook.py
[x] airflow/providers/slack/operators/slack.py

Provider smtp

[x] airflow/providers/smtp/hooks/smtp.py

Provider snowflake

[x] airflow/providers/snowflake/hooks/snowflake.py
[x] airflow/providers/snowflake/hooks/snowflake_sql_api.py
[x] airflow/providers/snowflake/operators/snowflake.py
[x] airflow/providers/snowflake/transfers/copy_into_snowflake.py
[x] airflow/providers/snowflake/triggers/snowflake_trigger.py
[x] airflow/providers/snowflake/utils/sql_api_generate_jwt.py

Provider sqlite

[x] airflow/providers/sqlite/hooks/sqlite.py

Provider ssh

[x] airflow/providers/ssh/hooks/ssh.py

Provider tableau

[x] airflow/providers/tableau/hooks/tableau.py
[x] airflow/providers/tableau/operators/tableau.py

Provider tabular

[x] airflow/providers/tabular/hooks/tabular.py

Provider telegram

[x] airflow/providers/telegram/hooks/telegram.py
[x] airflow/providers/telegram/operators/telegram.py

Provider trino

[x] airflow/providers/trino/hooks/trino.py

Provider weaviate

[x] airflow/providers/weaviate/hooks/weaviate.py

Provider yandex

[x] airflow/providers/yandex/hooks/yandex.py

Committer

[X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

guptarohit commented 7 months ago

I can help on these, if not all modules are picked or in progress.

Taragolis commented 7 months ago

@guptarohit feel free to pick up any providers or even core

rawwar commented 7 months ago

@Taragolis , What to do when the docstring doesn't actually explain anything about the method?

Example:

https://github.com/apache/airflow/blob/7835fd2659335d3acf830ce2e70dc19bfc5b2a84/airflow/providers/google/cloud/hooks/bigquery.py#L2350

Can i add noqa for docstrings like this?

EDIT:

slack message: https://apache-airflow.slack.com/archives/CCPRP7943/p1707489704051959?thread_ts=1707489399.332569&cid=CCPRP7943

possible rewrite : “Do nothing. Not needed for BigQueryConnection” ?

okirialbert commented 7 months ago

Hey, I'd like to work on any pending modules.

apache / airflow