Add etl module section - Githubissues

archesproject / arches-docs

official repo for Arches documentation

https://arches.readthedocs.io

9 stars 21 forks source link

Add etl module section #391

Closed njkim closed 10 months ago

njkim commented 10 months ago

brief description of changes

Add the section about the etl module

This box must be checked

[x] the PR branch was originally made from the base branch

This box should be checked

[x] after these changes the docs build locally without error

This box should only be checked you intend to follow through on it (we can do it on our end too)

[ ] I will cherry-pick all commits in this PR into other branches that should have them after this PR is merged

ekansa commented 10 months ago

Thanks @njkim and @apeters ! I'll take a look after the requested changes get implemented.

ekansa commented 10 months ago

the only issue I see with this is that currently there is no way when registering a new etl module to run celery tasks. We'd need to edit tasks.py to achieve that. Maybe we need to make a generic celery task runner?

@apeters from what I read of your comment, there should be a code modification in Arches core to deal with task management for the ETL module? Just to clarify, do you think this PR is ready to go to update the docs?

(cc @njkim )

apeters commented 10 months ago

the only issue I see with this is that currently there is no way when registering a new etl module to run celery tasks. We'd need to edit tasks.py to achieve that. Maybe we need to make a generic celery task runner?

@apeters from what I read of your comment, there should be a code modification in Arches core to deal with task management for the ETL module? Just to clarify, do you think this PR is ready to go to update the docs?

(cc @njkim )

@ekansa I guess as far as documentation goes I think you're right. It's probably fine at the moment given there's a section for how to create a new task.

ekansa commented 10 months ago

OK! @njkim @apeters I will merge this now.