I/O Design Meta-Issue

[mppf]

This issue is just meant to list and track I/O issues.

Priority Group 1

• [ ] Improve Channel Formats #7846
• [ ] I/O: When should output be machine readable? #7952
• [x] I/O plugins should be written in Chapel #8315

Priority Group 2

• [ ] when should an array be resized when it is read? #5053 (also #7524)
• [x] error handling: returning errors from writeThis #7261

Priority Group 3

• [ ] buffer and bytes should work with channel #8314
• [ ] some methods on channel begin with _ and we might consider renaming 'advance' to 'skip'
• [x] mark / _mark return an error code but should throw #13312
• [ ] check for I/O functions that would be more usable if they returned the result (since the return was typically used for error codes but now there are errors for that)
• [ ] allow passing DefaultRectangular to channel.readBytes`; possibly replace readBytes with something else.
• [ ] allow reading classes with allocation, instead of only reading into allocated classes.
• [ ] Supporting I/O of class instance cycles? #7951
• [ ] writeThis and inheritance #7118

Priority Group 4

• [ ] Remote file local channel #7953
• [ ] parallel file.lines() #4959
• [x] Spinwaiting in I/O lock can significantly degrade performance under qthreads #7748 (not expected to impact API)

[bradcray]

Related: @benharsh has been looking at I/O overheads in the context of the revcomp benchmark and expects to put together a list of I/O issues / wishes in response to that exercise, as I understand it.

[benharsh]

Question: does anything in the release/examples/ directory rely on the functionality we intend to deprecate?

[benharsh]

My take on a rough prioritization of these tasks (grouped by coarse prioritization):

1. Improve/add channel formats Rationale: probably deprecates "%j", breaking existing code
2. machine readable output Rationale: behavior-changing, possibly significantly

3. IO plugins written in Chapel Rationale: deprecates HDFS, CURL from stdlib

4. Return errors from writeThis Rationale: Future programs may break if they do not handle errors.

5. Arrays resized when read (or, "how to read an unknown number of things") Rationale: Useful idiom. Note: May depend on whether we keep mutate-through-array functionality.

6. Buffer/bytes/strings with channels Rationale: Useful idiom.
7. Methods beginning with underscore Rationale: Solidify API
8. DefaultRectangular and readBytes Rationale: may break existing code
9. allocate-as-read classes Rationale: Useful idiom.
10. class instance cycles Rationale: Annoying for users

11. writeThis inheritance Rationale: Annoying default behavior, but can be overrided

12. Remote file, local channel Rationale: correctness, productivity > performance. Though probably a common idiom.
13. parallel file.lines() Rationale: correctness, productivity > performance. Though this may have design repercussions.
14. Spinwaiting under qthreads Rationale: correctness, productivity > performance

[mppf]

thanks @benharsh - I've reordered the issue description to reflect your suggestion.

[e-kayrakli]

It might be good to keep a record of code style issues with the IO module here. See https://github.com/chapel-lang/chapel/issues/14646

[lydia-duncan]

I recorded some documentation tasks related to IO on #14395, should I link them here as well?

[mppf]

I think this issue should focus on things that fit into the API stabilization topic. Also this particular issue has things to address in a sorted order, so it might not make sense to try to add a big variety of small things to it.

Perhaps linking other issues to the epic Stabilize core I/O routines #8639 is the right move.