Excluding samples numerically / by index

[arpanda]

Hi, Initially, I ran 10 samples using the below project setup.

looper:
  output_dir: /home/my_path
  cli:
    run:
      limit: 10

So for the next run, how can I skip those first 10 samples. can you please help. Thanks.

[stolarczyk]

I don't think there's a straightforward way to do this. You could try to use the selector or toggle features to restrict which samples should be included.

looper run -h

...

sample selection arguments:
  Specify samples to include or exclude based on sample attribute values

  -g K, --toggle-key K               Sample attribute specifying toggle. Default: toggle
  --sel-attr ATTR                    Attribute for sample exclusion OR inclusion
  --sel-excl [E ...]                 Exclude samples with these values
  --sel-incl [I ...]                 Include only samples with these values

that said, this has been proposed before and will be implemented soon: https://github.com/pepkit/looper/issues/321

[arpanda]

Instead of skip, range for samples may be easier for the user for the limit. Like 1-10 or 10-15.

[vreuter]

https://github.com/pepkit/looper/issues/321#issuecomment-807286641

it's only really helpful for things that don't speak pipestat, once that's finished.

[vreuter]

@nsheff @donaldcampbelljr should this be a "skip-first-N" as suggested in #321 or rather instead / also accept a range as suggested by @arpanda ?

[vreuter]

There's also a question of for which subcommands this option should be available. For something like runp it clearly makes no sense, but for something like destroy, it's open for discussion / decision, I think.

[nsheff]

In discussion with @donaldcampbelljr we think doing a range makes sense; this could apply to both --skip and --limit.

So,

• --skip 5 would skip the first 5; --skip 7-10 would run 1-6 and 11+.
• --limit 5 would run the first 5; --limit 7-10 would run only 7-10

[nsheff]

There's also a question of for which subcommands this option should be available. For something like runp it clearly makes no sense, but for something like destroy, it's open for discussion / decision, I think.

I think having --limit and --skip on destroy would be useful.

[vreuter]

Looking at the output of looper called by itself to re-acquaint with the full collection of subcommands available, I think the following are the ones for which this idea definitely (or at least probably) makes sense. @nsheff @donaldcampbelljr please feel free to advocate for others' addition or removal of one or more listed here if you see it differently.

• [x] run -- initial motivator
• [x] rerun
• [x] check
• [x] destroy -- confirmed by @nsheff
• [x] clean