grailbio / reflow

A language and runtime for distributed, incremental data processing in the cloud
Apache License 2.0
965 stars 52 forks source link

How to `sum` file sizes? #104

Closed olgabot closed 5 years ago

olgabot commented 5 years ago

Is there a way to sum multiple file sizes to assign memory, e.g. something like this: (this doesn't work)

func SetDist(sketches [file], threads int) ={
    mem := sum([len(sketch) | sketch <- sketches ])
    exec(image := dashing, mem := 10*mem, cpu := threads) (matrix file) {"
        dashing setdist -H -TO {{matrix}} \
            -p {{threads}} {{sketches}}
    "}
}
prasadgopal commented 5 years ago

Currently there is no way of doing this. I will send out a change today that can do this.

On Fri, Feb 8, 2019 at 2:36 PM Olga Botvinnik notifications@github.com wrote:

Is there a way to sum multiple file sizes to assign memory, e.g. something like this: (this doesn't work)

func SetDist(sketches [file], threads int) ={ mem := sum([len(sketch) | sketch <- sketches ]) exec(image := dashing, mem := 10*mem, cpu := threads) (matrix file) {" dashing setdist -H -TO {{matrix}} \ -p {{threads}} {{reads}} "} }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/grailbio/reflow/issues/104, or mute the thread https://github.com/notifications/unsubscribe-auth/AfC0Qz6iIWyDcHzExpOAmkcJD2AXs4umks5vLfvhgaJpZM4ax6bT .

--

This email message, including attachments, may contain private, proprietary, or privileged information and is the confidential information and/or property of GRAIL, Inc., and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

olgabot commented 5 years ago

Great, thank you!