JesusFilm / core

JFP DS Engineering Monorepo
https://docs.core.jesusfilm.org/
19 stars 12 forks source link

feat: api-analytics #2891

Closed tataihono closed 8 months ago

tataihono commented 8 months ago

Description

Issue

Sites API was removed from Plausible CE 2.1. This means that the plausible service is no longer able to create new sites on plausible for data collection.

Solution

Add new graphql API called api-analytics. This will serve as the new sites API and will directly write to plausible postgres database. As a new API I am taking the opportunity to use the new pothos js graphql schema builder alongside the yoga server. This is a major departure from our class based nest service. This is a small project to introduce the new pattern to devs in the future. @mikeallisonJS has built similar things for api-languages rewrite.

Since api-journeys is going to call the new api-analytics graphql API, typing should be added to it to ensure that apollo client is strongly typed. This required updates to the email service since that was using untyped graphql calls. This uses the all new @graphql-codegen/cli library instead of the now deprecated apollo codegen. This represents the future tool we will need to migrate all our codegen across to in future.

External Changes

New API deployment requiring following envs to be added to doppler:

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

github-actions[bot] commented 8 months ago
Warnings
:warning: :exclamation: Big PR (13494 changes)
:warning: Is this PR related to a Basecamp issue? If so link it via the PR description - https://3.basecamp.com/3105655/buckets/:projectId/todos/:todoId

(change count - 13494): Pull Request size seems relatively large. If Pull Request contains multiple changes, split each into separate PR will helps faster, easier review.

Generated by :no_entry_sign: dangerJS against 01eeabcc2418e170940d3d55d3dfd18d60734e1b

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

infracost[bot] commented 8 months ago

πŸ’° Infracost report

Monthly cost will increase by $72 πŸ“ˆ

Changed project Baseline cost Usage cost* Total change New monthly cost
JesusFilm/core/infrastructure +$72 +$0 +$72 (+7%) $1,102

*Usage costs can be estimated by updating Infracost Cloud settings, see docs for other options.

Cost details ``` Key: * usage cost, ~ changed, + added, - removed ────────────────────────────────── Project: infrastructure Module path: infrastructure + module.prod.module.api-analytics.module.ecs-task.aws_ecs_service.ecs_service +$36 + Per GB per hour +$6 + Per vCPU per hour +$30 + module.stage.module.api-analytics.module.ecs-task.aws_ecs_service.ecs_service +$36 + Per GB per hour +$6 + Per vCPU per hour +$30 + module.prod.module.api-analytics.module.ecs-task.aws_cloudwatch_log_group.ecs_cw_log_group Monthly cost depends on usage + Data ingested Monthly cost depends on usage +$0.50 per GB + Archival Storage Monthly cost depends on usage +$0.03 per GB + Insights queries data scanned Monthly cost depends on usage +$0.005 per GB + module.prod.module.api-analytics.module.ecs-task.aws_ecr_repository.ecr_repository Monthly cost depends on usage + Storage Monthly cost depends on usage +$0.10 per GB + module.prod.module.api-analytics.module.ecs-task.aws_route53_record.record Monthly cost depends on usage + Standard queries (first 1B) Monthly cost depends on usage +$0.40 per 1M queries + Latency based routing queries (first 1B) Monthly cost depends on usage +$0.60 per 1M queries + Geo DNS queries (first 1B) Monthly cost depends on usage +$0.70 per 1M queries + module.stage.module.api-analytics.module.ecs-task.aws_cloudwatch_log_group.ecs_cw_log_group Monthly cost depends on usage + Data ingested Monthly cost depends on usage +$0.50 per GB + Archival Storage Monthly cost depends on usage +$0.03 per GB + Insights queries data scanned Monthly cost depends on usage +$0.005 per GB + module.stage.module.api-analytics.module.ecs-task.aws_ecr_repository.ecr_repository Monthly cost depends on usage + Storage Monthly cost depends on usage +$0.10 per GB + module.stage.module.api-analytics.module.ecs-task.aws_route53_record.record Monthly cost depends on usage + Standard queries (first 1B) Monthly cost depends on usage +$0.40 per 1M queries + Latency based routing queries (first 1B) Monthly cost depends on usage +$0.60 per 1M queries + Geo DNS queries (first 1B) Monthly cost depends on usage +$0.70 per 1M queries Monthly cost change for JesusFilm/core/infrastructure (Module path: infrastructure) Amount: +$72 ($1,030 β†’ $1,102) Percent: +7% ────────────────────────────────── Key: * usage cost, ~ changed, + added, - removed 8 projects have no cost estimate changes. Run the following command to see their breakdown: infracost breakdown --path=/path/to/code ────────────────────────────────── *Usage costs can be estimated by updating Infracost Cloud settings, see docs for other options. 823 cloud resources were detected: βˆ™ 209 were estimated βˆ™ 614 were free Infracost estimate: Monthly cost will increase by $72 ↑ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Changed project ┃ Baseline cost ┃ Usage cost ┃ Total change ┃ ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━╋━━━━━━━━━━━━╋━━━━━━━━━━━━━━┫ ┃ JesusFilm/core/infrastructure ┃ +$72 ┃ +$0 ┃ +$72 (+7%) ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━━━━┛ ```

This comment will be updated when code changes.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

stage-branch-merger[bot] commented 8 months ago

I see you added the "on stage" label, I'll get this merged to the stage branch!

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
docs βœ… Ready Visit Preview Thu Jun 20 00:51:04 UTC 2024
github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
arclight βœ… Ready Visit Preview Thu Jun 20 00:51:41 UTC 2024
github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
nexus-admin βœ… Ready Visit Preview Thu Jun 20 00:51:16 UTC 2024
github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
journeys βœ… Ready Visit Preview Thu Jun 20 00:52:23 UTC 2024
github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
watch βœ… Ready Visit Preview Thu Jun 20 00:53:22 UTC 2024
github-actions[bot] commented 8 months ago

The latest updates on your projects.

Name Status Preview Updated (UTC)
journeys-admin βœ… Ready Visit Preview Thu Jun 20 00:55:05 UTC 2024
jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.

codecov[bot] commented 8 months ago

Codecov Report

Attention: Patch coverage is 83.66013% with 25 lines in your changes missing coverage. Please review.

Project coverage is 88.77%. Comparing base (f8512d1) to head (01eeabc). Report is 1277 commits behind head on main.

Files with missing lines Patch % Lines
...api-journeys/src/__generated__/fragment-masking.ts 27.27% 8 Missing :warning:
...i-journeys/src/app/modules/email/email.consumer.ts 61.53% 4 Missing and 1 partial :warning:
apps/api-analytics/src/schema/generate.ts 66.66% 3 Missing :warning:
apps/api-analytics/src/yoga.ts 76.92% 2 Missing and 1 partial :warning:
apps/api-analytics/src/schema/index.ts 50.00% 1 Missing and 1 partial :warning:
apps/api-analytics/src/__generated__/gql.ts 75.00% 0 Missing and 1 partial :warning:
...i-analytics/src/schema/site/siteCreate.mutation.ts 93.75% 1 Missing :warning:
apps/api-analytics/test/timers.ts 87.50% 0 Missing and 1 partial :warning:
apps/api-journeys/src/__generated__/gql.ts 75.00% 0 Missing and 1 partial :warning:

:exclamation: There is a different number of reports uploaded between BASE (f8512d1) and HEAD (01eeabc). Click for more details.

HEAD has 3 uploads less than BASE | Flag | BASE (f8512d1) | HEAD (01eeabc) | |------|------|------| |apps.api-videos|6|3|
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2891 +/- ## ========================================== - Coverage 95.48% 88.77% -6.71% ========================================== Files 15 873 +858 Lines 354 14100 +13746 Branches 51 1871 +1820 ========================================== + Hits 338 12517 +12179 - Misses 10 1353 +1343 - Partials 6 230 +224 ``` | [Flag](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | Coverage Ξ” | | |---|---|---| | [apps.api-analytics](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `96.70% <90.67%> (?)` | | | [apps.api-journeys](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `96.13% <60.00%> (?)` | | | [apps.api-languages](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `82.14% <ΓΈ> (?)` | | | [apps.api-media](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `92.49% <ΓΈ> (?)` | | | [apps.api-tags](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `89.28% <ΓΈ> (?)` | | | [apps.api-users](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `81.59% <ΓΈ> (?)` | | | [apps.api-videos](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `95.48% <ΓΈ> (ΓΈ)` | | | [apps.journeys](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `44.58% <ΓΈ> (?)` | | | [apps.journeys-admin](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `81.03% <ΓΈ> (?)` | | | [apps.watch](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `76.29% <ΓΈ> (?)` | | | [libs.journeys.ui](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `78.13% <ΓΈ> (?)` | | | [libs.nest.common](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `86.36% <ΓΈ> (?)` | | | [libs.nest.decorators](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `100.00% <ΓΈ> (?)` | | | [libs.nest.powerBi](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `92.68% <ΓΈ> (?)` | | | [libs.shared.ui](https://app.codecov.io/gh/JesusFilm/core/pull/2891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm) | `83.01% <ΓΈ> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JesusFilm#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jesus-film-bot commented 8 months ago

Ran Plan for dir: infrastructure workspace: default

Plan Failed: This project is currently locked by an unapplied plan from pull #2851. To continue, delete the lock from #2851 or apply that plan and merge the pull request.

Once the lock is released, comment atlantis plan here to re-plan.