cmutel
commented
6 years ago This is a pretty big change - arrays should be treated the same way as in numpy.load, i.e. we can pass in an array, a string (or path), or a file-like object. I propose we punt on this for now and think about what we really need. Here are the constraints I see:
- We probably don't want to/can't copy arrays which are not in memory. But this means that we need to be able to support connections to non-local resources.
- Storing connections to non-local resources is tricky; we need to be able to store the metadata used to connect in JSON, so no python objects. There are probably many libraries for talking to different cloud providers.
- This adds complexity to both the code and our testing procedures
On the other hand, we do want to be able to use large presampled arrays generated by e.g. ecoinvent. The ideal case would be to support a library that wrapped all the complexity for us; I haven't found such a thing yet. Alternatively, we could only support e.g. AWS for now.
Current implementation requires that the numpy array is actually passed as an argument to
create_presamples_package. This therefore does not allow the use of an external address in the fielddatapackage['resources']['path'].