chrthomsen / pygrametl

Official repository for pygrametl - ETL programming in Python
http://pygrametl.org
BSD 2-Clause "Simplified" License
289 stars 41 forks source link

Add Windows support for decoupled BulkFactTable #47

Closed fromm1990 closed 1 year ago

fromm1990 commented 2 years ago

Currently, pygrametl does not support decoupled BulkFactTables on Windows. This is due to the filehandler passed to the bulkloader function cannot be pickled on Windows.

The code down below should reproduce the exception if run on a Windows machine.

import sqlite3

from pygrametl import ConnectionWrapper
from pygrametl.tables import BulkFactTable, DecoupledFactTable

connection = sqlite3.connect(":memory:")
cw = ConnectionWrapper(connection)

def bulk_loader(
    name: str, attributes, fieldsep, rowsep, nullval, filehandle
):
    pass

fact = BulkFactTable(name="test_table", keyrefs=["ra", "rb"], measures=["ma", "mb"], bulkloader=bulk_loader)
dfact = DecoupledFactTable(facttbl=fact) # <-- Produces TypeError: cannot pickle '_io.BufferedRandom' object on windows

A potential fix, could be postponing the creation of the tempdest field from the init function found in _BaseBulkloadable. More specifically, tempdest could be a property which is created on the first call, such the need of pickling is avoided.

skejserjensen commented 1 year ago

Closed by #55.