design discussion 2: N-D tensor specific things should in its own nimble package, on which other things (eg neural network functions etc) depend

[timotheecour]

/cc @mratsim N-D tensor specific things is much more limited in scope than, say, neural network stuff (which can grow in scope pretty much infinitely) a better design would be to factor out the N-D tensor specifics into a separate nimble package (say, ndtensor). Where to draw the line is debatable but for a lot of things this should be clear.

happy to discuss this in more details if discussion is welcome

in scope for ndtensor:

• creating a ndtensor from a size or from a view over an existing buffer
• slicing, concatenation, iteration, reshaping
• any operation that preserves data and just changes the view
• map, filter
• basic stringification, eg $

out of scope for ndtensor, ie move to other nimble package that depends on ndtensor

• resizing (note: different from reshaping), sacling, rotation and other image transforms
• image reading/writing
• anything computer vision or machine learning related
• convolution
• display
• linear algebra

[mratsim]

Yes in the long run I am considering it but for the moment the monolith approach is much easier to maintain.

It also seems to work well for PyTorch, Tensorflow and MxNet even at scale.

Another option that I kind of started doing is to use Nim submodules so that people can cherrypick for example:

import arraymancer/[tensor, ml]

to not import the NN part.

Resources:

• HN: Advantages of Monolitic Version Control (from danluu.com engineer@Google), several links at the bottom of the blog post.
• HN: On Monolithic Repositories, blog link dead but insightful comments