benschuler
commented
10 months ago Hi @agcopley,
Thanks for reaching out. When you run the deployment with the reporting objects configured as tables, the Cloud Build job will create the necessary DAG files and place them in the bucket that was specified in the config.json file as parameter targetBucket. When you look at the bucket, it should have a folder called dags and inside it various folders depending on the deployed workloads like sap or sfdc. Depending on the workloads that you have deployed, there can be a larger number of files and folders that are created in this bucket, so you might need to navigate to a second or third page when looking at your bucket content until the files or folders show up.
After the deployment is completed and you have created your Composer environment, you need to copy the DAG files from the output bucket of the deployment (the one mentioned above) to the DAG bucket of the composer environment. You can find a sample command in our readme file, simply replace the placeholders with your own values: https://github.com/GoogleCloudPlatform/cortex-data-foundation?tab=readme-ov-file#move-the-files-into-the-dag-bucket
If you are still having trouble locating the files, could you please share the content of your config.json file, the command that you used to run the deployment as well as screenshots of the Cloud Build history and the Cloud Storage bucket where you would expect the files to show up after deployment so we can take a closer look at what might be the issue here.
agcopley
Hello, I have deployed cortex 5.3 with the majority of the reporting objects as tables, implying that they are to be refreshed periodically by DAGS en Composer (according to the documentation). However I cant find the DAGS for the reporting objects in my specified bucket. What I am missing?
Many thanks!