Bug: MySQL tables without primary keys

[srust99]

Hello,

Our systems have indicated that your MySQL cluster, db-mysql-sfo3-53387, has tables without primary keys. We have identified that MySQL tables without primary keys can lead to service replication issues that jeopardize performance and availability. If primary keys are not present for database tables exceeding 5,000 rows, data-loss can occur.

We urge you to take the following actions to ensure that your database db-mysql-sfo3-53387 continues to operate correctly:

1. Review the following product documentation which details the issue and provides clear mitigation strategies for all impacted users.
2. Implement primary keys for all database tables as described in the documentation.

See the full details at the following link: https://www.digitalocean.com/docs/databases/mysql/how-to/create-primary-keys/

We will continue to notify you of this issue until all MySQL tables have primary keys.

DigitalOcean

Blocked by:

• https://github.com/truflation/website/issues/750
• https://github.com/truflation/website/issues/867

Problems

• [ ] https://github.com/truflation/truflation-ingestors/issues/220

[zolotokrylin]

@truflation/team-python-pipelines guys, please investigate and fix. Thank you!

[zolotokrylin]

@joequant @ambahk ?

[joequant]

This isnt an issue in production because we are not using hosted mysql. @ambahk is moving us to hosted mysql and the errors are coming from his test instances. Adding primary keys to our tables is a todo before we move our system to hosted mysql.

[Jarryd-pretorius]

Hey @joequant can you add me as an admin to the digital ocean please

[joequant]

Hey @joequant can you add me as an admin to the digital ocean please

Done. Just upgraded you to admin and you should get an confirm email to click

[Jarryd-pretorius]

Hey @georgeciubotaru Can you please look at this ticket first. I have added you to digital ocean let me know if you need any help or access

[georgeciubotaru]

Log: Based on the instructions from the link we should manually alter the current tables, and set up Primary key creations for the new ones.

Based on the query we have around 1060 tables to define primary keys

[georgeciubotaru]

@Jarryd-pretorius I'm trying to find where is the starting point for these tables, can you help me to accelerate that?

https://github.com/truflation/truflation/tree/6f4a7b995cae3b8c6dc8240a21c137550c0d382c/src/truflation

cc @joequant, @ambahk

[Jarryd-pretorius]

@joequant can you please give @georgeciubotaru the info he needs to solve this problem regarding the db

[joequant]

@Jarryd-pretorius I'm trying to find where is the starting point for these tables, can you help me to accelerate that?

https://github.com/truflation/truflation/tree/6f4a7b995cae3b8c6dc8240a21c137550c0d382c/src/truflation

cc @joequant, @ambahk

Can you telegram me on joequant. The easiest thing that I can I do is to give you access to the test database. @georgeciubotaru

[joequant]

@Jarryd-pretorius I'm trying to find where is the starting point for these tables, can you help me to accelerate that? https://github.com/truflation/truflation/tree/6f4a7b995cae3b8c6dc8240a21c137550c0d382c/src/truflation cc @joequant, @ambahk

Can you telegram me on joequant. The easiest thing that I can I do is to give you access to the test database. @georgeciubotaru

actually I'm at joequanthk

[georgeciubotaru]

Log: clarified during telegram chat.

Going to split this task into 2 sub-tasks:

• Update the codebase so the new tables will have primary keys and indexes similar to (https://github.com/truflation/truflation-ingestors/issues/220)
• Update the codebase so the existing tables will get primary keys and indexes if none of them exists

Context: Reference source

@Jarryd-pretorius FYI

[Jarryd-pretorius]

Awesome thanks @georgeciubotaru for the update

[Jarryd-pretorius]

Hey @georgeciubotaru how is the progress on this?

[georgeciubotaru]

Hey @Jarryd-pretorius, I am currently trying to test the changes

[markholdex]

@georgeciubotaru any ETA?

[georgeciubotaru]

@markholdex Still haven't a successful result for testing, but the ETA is EOD.

[georgeciubotaru]

Log: Requested review for the PR: https://github.com/truflation/truflation/pull/49

if no major changes requested the ETA is EOD

[markholdex]

@Jarryd-pretorius can you please help follow up Joseph here? I know he might be afraid of something breaking on production but with a bit of attention to the review, it can be mitigated. Let me know if anything changes about this Goal in favor of:

• https://github.com/truflation/truflation-transformers/issues/180

p.s. I know George is working on it while waiting review but still asking.

[Jarryd-pretorius]

Yeah I feel the same way I pinged @joequant yesterday about getting this reviewed I will follow this up again right now on TG @markholdex

[markholdex]

log: moving back into Design Ready to not confuse everybody until the Goal is not blocked anymore.

[Jarryd-pretorius]

@georgeciubotaru where are we at with this can we merge this in now?

[georgeciubotaru]

@Jarryd-pretorius This task is on hold as wasn't fully tested on how it will interact with the production data.

We definitely need to resolve our process issues and we can incorporate that fix

[Jarryd-pretorius]

We definitely need to resolve our process issues and we can incorporate that fix

Ok right what needs to be done process wise to handle this?

[markholdex]

log: discussed with George. He will pick up this Problem himself. Needs to be super careful with it as it may impact production negatively.

[zolotokrylin]

@georgeciubotaru please fix this one after you finish with:

• https://github.com/truflation/website/issues/926
• https://github.com/truflation/truflation-ingestors/issues/292