Open Stochastic-Squirrel opened 8 months ago
@Stochastic-Squirrel Thanks for reporting this issue. The team is taking a look and we'll get back to you shortly
Hi @Stochastic-Squirrel
We're having some difficulty in reproducing the issue. We'll continue to look into this.
It looks like you already ran PipeRider with --debug
? If not, I would suggest that as a further debugging step. There may be a lot of output - If there's nothing sensitive in the output you could share that with us. Either attach it here, or email to product@piperider.io
In the meantime, you could try out an Impact Report manually by using the DBT Manifest Analyzer in PipeRider Cloud:
I'll follow up on this when we have had more success reproducing the issue.
Thanks,
Dave
Hi Dave, thanks for reaching out!
I have been using the --debug
flag throughout and nothing is printed to the console. Unfortunately I haven't been able to glean any more info!
I tried the website now and I experienced a server error when attempting uploads
Sentry event id: 229c6e399c2b48d08f61682dff0ac69a
I have tried uploading a single manifest as well as two at the same time.
Unfortunately I don't feel comfortable sharing the manifests in their entirety. I'll try to cut them down to a minimal set. What are the essential keys needed? I am thinking of maybe isolating a single table that does not contain any sensitive information.
However, I do have an update on what causes the error! I experimented some more and I noticed that it is only when changing the model YAML files that the error occurs. Changes to the SQL models seem to work fine.
Here are some scenarios that causes the error
array<string>
to array
fruit
to fruits_and_vegetables
in staging in the model yamlIn my case, I have to tweak model yamls for SQL models that are affected by a change e.g. data type change.
I hope this makes it a bit easier to recreate the issue on your end.
@Stochastic-Squirrel
Hi, thanks for your information. I know there is some privacy concern for providing the real run.json
. So Is it possible to provide the two run.json
from a dummy project? It would help us to reproduce the issue.
piperider compare-reports --base run_base.json --target run.json --output ./comparison_report
Selected reports:
Base: run_base.json
Target: run.json
Warning:
'bigquery'
Got problem to generate changeset.
Comparison report:
/data_warehouse/comparison_report/index.html
thanks @popcornylu. I'll try to reproduce this error in a dummy project as soon as I can!
First of all, I am really enjoying this tool! Unfortunately I have come across this bug which is blocking a rollout to the wider team so I am hoping that there is a quick fix!
Describe the bug When comparing two piperider reports, a warning "bigquery" is returned and no comparison summary is generated.
Reproduce Unfortunately, I cannot provide the manifest jsons but I will try my best to describe the issue and steps taken.
What's strange is that the diff summary report works for some MRs but not others. I have tried to find the common trait but I am unable to. The MR and subsequent report comparison that works is a very simple test case where I add a text column to an existing table with a constant value e.g.
Looking at the comparison report, row and col information for both base (production) and target (staging) are recorded.
What I have tried
piperider diagnose
passesI attached a debugger and I tried to figure out what was going on.
GraphDataChangeSet
object fails to be createdlist_changes_in_unique_id
failingbigquery
(hence the warning printed on the console), theAdapterContainer
'slookup_adapter
function is called which attempts to extract thebigquery
adapter using thebigquery
keyRelevant code linked here
Expected behavior Diff summary reports for dbt models that have been changed. Example output below from the successful MR comparison
Desktop (please complete the following information):