search

chrthomsen / pygrametl

Official repository for pygrametl - ETL programming in Python
http://pygrametl.org
BSD 2-Clause "Simplified" License
289 stars 41 forks source link

_BaseBulkloadable only resolves dependencies on first bulk run #72

Open fromm1990 opened 2 weeks ago

fromm1990 commented 2 weeks ago

Due to the filter on this line dependson only resolves once. The built-in filter function in python returns an iterator. Therefore, the dependson attribute can only be iterated once, hence all concecutive batchloads will not force dependencies to load.

A quick fix would be the following 

self.dependson = filter(lambda b: hasattr(b, '_bulkloadnow'), dependson)
self.dependson = list(self.dependson)

or

self.dependson = [d for d in dependson if hasattr(d, '_bulkloadnow')]

Either of the two solutions results in a list which alows for multiple iterations.

chrthomsen commented 1 week ago

Hi,

Thanks! Good catch. I agree with the proposed solutions (with a slight preference for the 2nd).

Best regards, Christian

fromm1990 commented 1 week ago

I have created a PR #73