stripe-archive / mosql

MongoDB → PostgreSQL streaming replication
MIT License
1.63k stars 225 forks source link

convert binary data to hex. fixes #64. #65

Closed spazm closed 10 years ago

spazm commented 10 years ago

Fixes UTF8 errors by encoding binary data in hex.

PG::CharacterNotInRepertoire: ERROR:  invalid byte sequence
 for encoding "UTF8": 0xaf (Sequel::DatabaseError)

Note, BSON::DBRef may need a similar treatment in lines 186 and 190?

Verification:

I have used this to import UUID from mongo to postgresql

> db.entities.findOne();
{
        "_id" : BinData(3,"DHSvnWWKRQ6hydIk/B4RaQ=="),
        "slug" : "DHSvnWWKRQ6hydIk_B4RaQ"
}

collections.yml:

    db2:
      entities:
        :columns:
        - id:
          :source: _id
          :type: UUID
        - slug: TEXT
        :meta:
          :table: entities
          :extra_props: false
vagrant=# select * from entities where slug='DHSvnWWKRQ6hydIk_B4RaQ'
;
                  id                  |          slug
--------------------------------------+------------------------
 0c74af9d-658a-450e-a1c9-d224fc1e1169 | DHSvnWWKRQ6hydIk_B4RaQ
(1 row)