larray-project / larray

N-dimensional labelled arrays in Python
https://larray.readthedocs.io/
GNU General Public License v3.0
8 stars 6 forks source link

implement nsplit argument in Array.split_axes with default="auto" (raise if variable) #1078

Open gdementen opened 1 year ago

gdementen commented 1 year ago

When the number of separators is not the same in all labels, labels which have more separators have their last part dropped silently and any label which becomes a duplicate because of this is dropped silently too (it seems like the last duplicate wins).

>>> arr = la.ndtest("a_b=a0_b0,a0_b1_1,a0_b1_2")
>>> arr
a_b  a0_b0  a0_b1_1  a0_b1_2
         0        1        2
>>> arr.split_axes()
a\b   b0   b1
 a0  0.0  2.0

Also, when there is a label with fewer separators than expected, it raises a very weird error.

>>> arr = la.ndtest("a_b=a0_b0,a0b1,a0_b2")
>>> arr
a_b  a0_b0  a0b1  a0_b2
         0     1      2
>>> arr.split_axes()
ValueError: Value {a_b} axis is not present in target subset {a*}. A value can only have the same axes or fewer axes than the subset being targeted

The two issues are because the zip(*sequences) idiom that we use happily ignores sequences longer than the shortest of the sequences.

The code of Axis.split is roughly like this:

>>> split_labels = np.char.split(arr.a_b.labels, '_')
>>> indexing_labels = zip(*split_labels)
>>> list(indexing_labels)
[('a0', 'a0', 'a0'), ('b0', 'b1', 'b1')]
>>> split_axes = [Axis(unique_list(ax_labels), name) for ax_labels, name in zip(indexing_labels, names)]

Our options are :

  1. raise an error if fewer or too many separators detected. Safest (zero chance to have bad labels pass unnoticed) but not practical (it leaves the problem to the user)
  2. use maxsplit=len(names) - 1 in np.char.split. This would solve the extra separators issue but not the fewer separators issues. Also it could mask a problem if there are extra seps in some labels that the user does not know about. I can live with the later problem though.
  3. or configurable maxsplit, defaulting to the above. More flexible, but same problems as option 1.
  4. or force splitting using the maximum number of separators (i.e. complete the sequences with as many '' as necessary). In the presence of too many labels, it will probably not be the result the user wants in most cases.
  5. force splitting using len(names) - 1 (i.e use maxsplit but then complete the too short sequences)
  6. configurable force splitting (e.g. nsplit argument), using len(names) - 1 as default.
  7. configurable force splitting , but with a default which raises if not all sequence have the same length. I hesitate with this option. The selected option
gdementen commented 1 year ago

@alixdamman : what do you think?

alixdamman commented 1 year ago
  1. raise an error if fewer or too many separators detected. Safest (zero chance to have bad labels pass unnoticed) but not practical (it leaves the problem to the user)
  2. use maxsplit=len(names) - 1 in np.char.split. This would solve the extra separators issue but not the fewer separators issues. Also it could mask a problem if there are extra seps in some labels that the user does not know about. I can live with the later problem though.
  3. or configurable maxsplit, defaulting to the above. More flexible, but same problems as option 1.
  4. or force splitting using the maximum number of separators (i.e. complete the sequences with as many '' as necessary). In the presence of too many labels, it will probably not be the result the user wants in most cases.
  5. force splitting using len(names) - 1 (i.e use maxsplit but then complete the too short sequences)
  6. configurable force splitting (e.g. nsplit argument), using len(names) - 1 as default. My preferred solution at this point.
  7. configurable force splitting , but with a default which raises if not all sequence have the same length. I hesitate with this option.

Honestly, without an example of the resulting axes for each option, it is difficult to me to evaluate what would be the best solution. I am usually in favor of somewhat "forcing" the users to clean their code (i.e. check the labels, dimensions of arrays, ...), so, by default, my answer is option 1. But if you think that there are potentially many cases in which option 1 could not be practical, I am OK to choose the option 6.

gdementen commented 1 year ago

In that case, I'd go for option 7, which is basically option 1 with an easy way to deal with the exception when it happens. I will try to implement that.

gdementen commented 1 year ago

Implementing nsplit is not that urgent. Avoiding the silent failure is the urgent part so I extracted #1089 out of this and scaled down the priority.

gdementen commented 1 year ago

Besides, to be complete, we should implement a way to split right (rsplit boolean argument)?