Qovery / Replibyte

Seed your development database with real data ⚡️
https://www.replibyte.com
GNU General Public License v3.0
4.17k stars 129 forks source link

Running replibyte results in command being killed #244

Open pKorsholm opened 1 year ago

pKorsholm commented 1 year ago

I'm currently running replibyte trying to subset my database.

My config is something like:

source:
  connection_uri: $DB_URL
  database_subset:
    database: public
    table: cart
    strategy_name: random
    strategy_options:
      percent: 20
    passthrough_tables: 
      - country
      .... 
      <30 more tables> 
  transformers: 
    - database: public
      table: customer
      columns:
        - name: email
          transformer_name: email
    ... 
    <23 columns and 8 tables>    

This results in replibyte running out of memory (I'm guessing given the screenshot of the process from my activity montitor below, taken right before the process crashed)

image

Is there any way to reduce the memory consumption, the database is in total around 250mb (148mb if asking postgres with: SELECT pg_size_pretty( pg_database_size('dbname') )) and 51gb of memory taken up seems excessive.