altaurog / pgcopy

fast data loading with binary copy
https://pgcopy.readthedocs.io/
Other
113 stars 33 forks source link

add numeric datatype #1

Closed koshak01 closed 7 years ago

koshak01 commented 7 years ago
def split_by(number, length = 4):
    data = [ int(number[-length-once:len(number) - once]) for once in range(0, len(number), length)]
    data.reverse()
    return data

def numeric_formater(val):
    val = str(val)
    try:
        dots = val.index('.')
        data_base = [(2, 'h', once) for once in split_by(val[:dots], 4)]
        dots_based = "{0}{1}".format(val[dots+1:], '0'*(4 - len(val[dots+1:])%4))
        data_dots = [(2, 'h', once) for once in split_by(dots_based, 4)]
        data = data_base + data_dots
        dots_positions = len(val) - dots - 1
    except ValueError:
        data = [(2, 'h', once) for once in split_by(val, 4)]
        data_base = data
        # dots = None
        dots_positions = None

    header = [
         (2, 'h', len(data))
        ,(2, 'h', len(data_base) - 1)
        ,(4, 'i', 0 if not dots_positions else dots_positions)
    ]
    data = header + data
    count = sum([once[0] for once in data])
    data.insert(0, (4, 'i', count))
    format = ''.join([once[1] for once in data])
    return (format, [once[2] for once in data] )

take from https://doxygen.postgresql.org/backend_2utils_2adt_2numeric_8c.html#a57a8f8ab552bae24926d252180956958

altaurog commented 7 years ago

Hey, thanks for your interest, @koshak01! numeric would be a great addition to the library, but it would need a more complete implementation. I'll have a shot at it when time permits. What are your thoughts about the type of the python val? Would it serialize any type, or just decimal.Decimal?

filippzorin commented 2 years ago

Hey, thanks for your interest, @koshak01! numeric would be a great addition to the library, but it would need a more complete implementation. I'll have a shot at it when time permits. What are your thoughts about the type of the python val? Would it serialize any type, or just decimal.Decimal?

Hi @altaurog, I'm just started using pgcopy and faced with issue when insert int values to postgres numeric. Why you decided to serialize only decimal.Decimal? I think it would be good if at least int can be serialized to numeric. In my opinion, there is no any corner cases with int, not sure about other types which can be converted casted to decimal.Decimal, this is also discussable. What do you think about this?

altaurog commented 1 year ago

Hey @filippzorin, I am glad you have taken interest in this project.

I think the primary motivation was a simple and natural mapping between types. It is true that int can be serialized to numeric, but int can easily be converted first to decimal.Decimal as well. We could add some magic in the library to do that automatically for convenience, but there would have to be a compelling motivation. An int in python will normally be stored in the database as int.