AWS handlers that are not reference are not registered

[eugeneteoh]

Describe the bug

I encountered an error when trying to use the GraphQL query, getSqlPipelineFileContent.

How to Reproduce

query test {
  getSqlPipelineFileContent(input: { 
    sqlPipelineUri: "wc9s7pc7", # ANY VALID SQL PIPELINE URI
    absolutePath: "version.txt" # ANY VALID FILE,
    branch: "main"
  })
}

Expected behavior

dataall-graphql-1        |  found handler None for task action repo.sqlpipeline.cat|pwe1fwjy
dataall-graphql-1        | ==================> No handler defined for repo.sqlpipeline.cat
dataall-graphql-1        | Error in process
dataall-graphql-1        | Traceback (most recent call last):
dataall-graphql-1        |   File "/code/dataall/aws/handlers/service_handlers.py", line 42, in process
dataall-graphql-1        |     handler, task = self.get_task_handler(engine, taskid)
dataall-graphql-1        |   File "/code/dataall/aws/handlers/service_handlers.py", line 81, in get_task_handler
dataall-graphql-1        |     raise Exception(f'No handler defined for {task.action}')
dataall-graphql-1        | Exception: No handler defined for repo.sqlpipeline.cat
dataall-graphql-1        | Task processing failed No handler defined for repo.sqlpipeline.cat : pwe1fwjy
dataall-graphql-1        | 'NoneType' object is not subscriptable
dataall-graphql-1        |
dataall-graphql-1        | GraphQL request:2:3
dataall-graphql-1        | 1 | query test {
dataall-graphql-1        | 2 |   getSqlPipelineFileContent(input: {sqlPipelineUri: "wc9s7pc7", absolutePath: "version.txt", branch: "main"})
dataall-graphql-1        |   |   ^
dataall-graphql-1        | 3 | }
dataall-graphql-1        | Traceback (most recent call last):
dataall-graphql-1        |   File "/usr/local/lib/python3.8/site-packages/graphql/execution/execute.py", line 521, in execute_field
dataall-graphql-1        |     result = resolve_fn(source, info, **args)
dataall-graphql-1        |   File "/code/dataall/api/Objects/__init__.py", line 89, in adapted
dataall-graphql-1        |     response = resolver(
dataall-graphql-1        |   File "/code/dataall/api/Objects/SqlPipeline/resolvers.py", line 147, in cat
dataall-graphql-1        |     return response[0]['response'].decode('ascii')
dataall-graphql-1        | TypeError: 'NoneType' object is not subscriptable
dataall-graphql-1        |
dataall-graphql-1        | The above exception was the direct cause of the following exception:
dataall-graphql-1        |
dataall-graphql-1        | Traceback (most recent call last):
dataall-graphql-1        |   File "/usr/local/lib/python3.8/site-packages/graphql/execution/execute.py", line 521, in execute_field
dataall-graphql-1        |     result = resolve_fn(source, info, **args)
dataall-graphql-1        |   File "/code/dataall/api/Objects/__init__.py", line 89, in adapted
dataall-graphql-1        |     response = resolver(
dataall-graphql-1        |   File "/code/dataall/api/Objects/SqlPipeline/resolvers.py", line 147, in cat
dataall-graphql-1        |     return response[0]['response'].decode('ascii')
dataall-graphql-1        | graphql.error.graphql_error.GraphQLError: 'NoneType' object is not subscriptable
dataall-graphql-1        |
dataall-graphql-1        | GraphQL request:2:3
dataall-graphql-1        | 1 | query test {
dataall-graphql-1        | 2 |   getSqlPipelineFileContent(input: {sqlPipelineUri: "wc9s7pc7", absolutePath: "version.txt", branch: "main"})
dataall-graphql-1        |   |   ^
dataall-graphql-1        | 3 | }

I digged deeper and found that the CodeCommit handler under backend/dataall/aws/handlers/codecommit.py was not registered to self.handlers under service_handlers.WorkerHandler. This is because codecommit.py is not reference anywhere else in the code, hence the decorator @Worker.handler is not executed at runtime. I attempted to print self.handlers and this is the output:

dict_keys(['glue.dataset.database.tables', 'glue.dataset.crawler.create', 'glue.crawler.start', 'glue.table.update_column', 'glue.table.columns', 'glue.job.runs', 'glue.job.start_profiling_run', 'glue.job.profiling_run_status', 'ecs.share.approve', 'ecs.share.reject', 'ecs.cdkproxy.deploy', 'cloudformation.stack.delete', 'cloudformation.stack.status', 'cloudformation.stack.describe_resources', 'environment.check.cdk.boostrap', 's3.prefix.create', 'redshift.cluster.init_database', 'redshift.cluster.create_external_schema', 'redshift.cluster.drop_external_schema', 'redshift.cluster.tag', 'redshift.iam_roles.update', 'redshift.subscriptions.copy'])

As you can see, the handlers in these files are not registered: codecommit.py, sns.py, sqs.py

Your project

No response

Screenshots

No response

OS

Mac

Python version

Python 3.8

AWS data.all version

-

Additional context

Quick Fix

I came up with a quick fix which is to add from . import codecommit to backend/dataall/aws/handlers/__init__.py. This is so that the file will be reference and compiled, so that the decorator @Worker.handler is executed. This is not the most elegant solution but I cannot think of a better solution at the moment.

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed.

[LEUNGUU]

I think maybe we can import it from where we add those tasks. PR will be raised.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.