techouse / sqlite3-to-mysql

Transfer data from SQLite to MySQL
https://techouse.github.io/sqlite3-to-mysql/
MIT License
371 stars 51 forks source link

I would like to be able to migrate the number of digits in numeric columns as well. #98

Closed sasaki000 closed 9 months ago

sasaki000 commented 9 months ago

Thank you for your continued support and development. The number of digits in numeric columns cannot be migrated correctly. In Django, I defined the trading_price column as a 16-digit integer part using DecimalField. Sqlite3 does not seem to have number of digits information. Therefore, the number of digits will be lost when migrating to MySQL. I would like it to be possible to migrate the number of digits by referencing the Django model class instead of from the db.sqlite3 file.

(1)The Django model class is below.

################################################################################
class JP_StockPriceMinuteModel(models.Model, MixinModel):
    """株価(週足)テーブル"""
    id = models.CharField(primary_key=True, max_length=21, verbose_name='ID')
    ticker = models.CharField(max_length=6, verbose_name='銘柄ID')
    date = models.DateTimeField(blank=False, null=False, verbose_name='日付(UTC)')
    open = models.DecimalField(max_digits=12, decimal_places=2, blank=True, null=True, verbose_name='始値')
    high = models.DecimalField(max_digits=12, decimal_places=2, blank=True, null=True, verbose_name='高値')
    low = models.DecimalField(max_digits=12, decimal_places=2, blank=True, null=True, verbose_name='安値')
    close = models.DecimalField(max_digits=12, decimal_places=2, blank=True, null=True, verbose_name='終値')
    volume = models.DecimalField(max_digits=16, decimal_places=2, blank=True, null=True, verbose_name='出来高')
    trading_price = models.DecimalField(max_digits=16, decimal_places=2, blank=True, null=True, verbose_name='売買代金')
    updated_at = models.DateTimeField(blank=True, null=True, verbose_name='更新日(UTC)')
    created_at = models.DateTimeField(blank=True, null=False, default=timezone.now, verbose_name='作成日(UTC)')

    def __str__(self):
        return f'{self.ticker}({pd.Timestamp(self.date).tz_convert(fin.Market.JP.value.tz)}) {self.open:,.2f} / {self.high:,.2f} / {self.low:,.2f} / {self.close:,.2f}'

    class Meta:
        db_table = 'orcas_jp_stock_price_minute'
        models.Index(fields=['ticker'], name=db_table+'_index')
        constraints = [
            models.UniqueConstraint(fields=['ticker', 'date'], name=db_table+'_unique')  # ユニーク制約
        ]
        verbose_name_plural = '[JP]株価(分足)'

(2)The table definition of the Sqlite3 database (db.sqlite3 file) is below.(Before migration)

(venv) sasaki@SasakiWin10Pro:~/workspace/orcas_proj/jupyter_proj/data$ sqlite3 db.sqlite3
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> pragma table_info(orcas_jp_stock_price_minute);
0|id|varchar(21)|1||1
1|ticker|varchar(6)|1||0
2|date|datetime|1||0
3|open|decimal|0||0
4|high|decimal|0||0
5|low|decimal|0||0
6|close|decimal|0||0
7|volume|decimal|0||0
8|trading_price|decimal|0||0
9|updated_at|datetime|0||0
10|created_at|datetime|1||0
sqlite>
sqlite> .exit

(3)The MySQL table definition is below.(After migration)

(venv) sasaki@SasakiWin10Pro:~/workspace/orcas_proj/jupyter_proj/data$ mysql -h 127.0.0.1 -P 3306 -u gpts -pXXXX
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 58
Server version: 8.0.30 MySQL Community Server - GPL

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use orcas;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> describe orcas_jp_stock_price_minute;
+---------------+---------------+------+-----+---------+-------+
| Field         | Type          | Null | Key | Default | Extra |
+---------------+---------------+------+-----+---------+-------+
| id            | varchar(21)   | NO   | PRI | NULL    |       |
| ticker        | varchar(6)    | NO   | MUL | NULL    |       |
| date          | datetime      | NO   |     | NULL    |       |
| open          | decimal(10,0) | YES  |     | NULL    |       |
| high          | decimal(10,0) | YES  |     | NULL    |       |
| low           | decimal(10,0) | YES  |     | NULL    |       |
| close         | decimal(10,0) | YES  |     | NULL    |       |
| volume        | decimal(10,0) | YES  |     | NULL    |       |
| trading_price | decimal(10,0) | YES  |     | NULL    |       |
| updated_at    | datetime      | YES  |     | NULL    |       |
| created_at    | datetime      | NO   |     | NULL    |       |
+---------------+---------------+------+-----+---------+-------+
11 rows in set (0.00 sec)

mysql> exit;
Bye
sasaki000 commented 9 months ago

I found that by first creating an empty table in MySQL using Django's ORM's python3 manage.py migrate command, I was able to migrate the data without losing any digits. Therefore, I think this request has a very low priority because it seems avoidable.

techouse commented 9 months ago

Hey!

You beat me to it. Was about to say that Django's ORM can already migrate schemas from database to database and then you can simply use this tool to migrate the data itself :)