Implement a function that calls a chain of subcommands piped into each other.

[LeeBergstrand]

          @jmtsuji Here, we might be able to implement a function that calls a chain of subcommands piped into each other.

Also, note that there may be more sophisticated libraries for performing this kind of work, like https://pypi.org/project/sh/, that remove some of the boilerplate. I was initially going to use sh for rotary, but there were some features that were missing so I did not use it. It still might be useful for your code.

_Originally posted by @LeeBergstrand in https://github.com/rotary-genomics/rotary-utils/pull/6#discussion_r1566571848_

[LeeBergstrand]

Previous Code

    with open(log_filepath, write_mode) as logfile_handle:
        with open(output_bam_filepath, 'w') as bam_handle:
            # TODO - add support for different flags like -ax for pacbio
            minimap_args = ['minimap2', '-t', str(threads), '-ax', 'map-ont', contig_filepath, long_read_filepath]
            minimap = run_pipeline_subcommand(command_args=minimap_args, stdout=subprocess.PIPE, stderr=logfile_handle)

            samtools_view_args = ['samtools', 'view', '-b', '-@', str(threads)]
            samtools_view = run_pipeline_subcommand(command_args=samtools_view_args, stdin=minimap,
                                                    stdout=subprocess.PIPE, stderr=logfile_handle)

            samtools_sort_args = ['samtools', 'sort', '-@', str(threads), '-m', f'{threads_mem_mb}M']
            run_pipeline_subcommand(command_args=samtools_sort_args, stdin=samtools_view, stdout=bam_handle,
                                    stderr=logfile_handle)

        samtools_index_args = ['samtools', 'index', '-@', str(threads), output_bam_filepath]
        run_pipeline_subcommand(command_args=samtools_index_args, stderr=logfile_handle)

The code above could be written as:

    with open(log_filepath, write_mode) as logfile_handle:
        with open(output_bam_filepath, 'w') as bam_handle:
            # TODO - add support for different flags like -ax for pacbio
            minimap_args = ['minimap2', '-t', str(threads), '-ax', 'map-ont', contig_filepath, long_read_filepath]
            samtools_view_args = ['samtools', 'view', '-b', '-@', str(threads)]
            samtools_sort_args = ['samtools', 'sort', '-@', str(threads), '-m', f'{threads_mem_mb}M']

            run_chained_subcommand(commands=[minimap_args, samtools_view_args, samtools_sort_args], stdout=bam_handle,  stderr=logfile_handle)

        samtools_index_args = ['samtools', 'index', '-@', str(threads), output_bam_filepath]
        run_pipeline_subcommand(command_args=samtools_index_args, stderr=logfile_handle)

[LeeBergstrand]

@jmtsuji Does this make sense to you? All you would need to do is have a function called run_chained_subcommand that loops through all the subcommand lists and pipes the output to the next subcommand in the chain. This removes all the calls to run_pipeline_subcommand, which are redundant, and cuts the code used in half.

[LeeBergstrand]

@jmtsuji This code could also be used for subset_reads_from_bam

[LeeBergstrand]

@jmtsuji I would recommend that we implement this before we implement any other external commands that use chained pipeing.

[jmtsuji]

@LeeBergstrand Thanks for making this issue and for the clarification. What you've proposed sounds quite feasible for adding a function that chains subcommands. I can revisit this and implement it before any other external commands with pipes are added. Also worthwhile to look into an alternative library like sh at that point.

[LeeBergstrand]

@LeeBergstrand Thanks for making this issue and for the clarification. What you've proposed sounds quite feasible for adding a function that chains subcommands. I can revisit this and implement it before any other external commands with pipes are added. Also worthwhile to look into an alternative library like sh at that point.

Thanks this sounds good to me!