TOCE as SIGCSE TS research track

[amyjko]

I wrote the SIGCSE steering committee

Hello illustrious SIGCSE TS steering committee,

I'm writing in my role as Editor-in-Chief of ACM Transactions on Computing Education, to see if there's interest in evolving the partnership between ACM TOCE and the symposium. This is a relatively long email, but it's a long term opportunity, so take your time to ruminate.

Currently, we have an agreement where ACM TOCE papers published prior to October the calendar year before the symposium are eligible to be presented at the next symposium. The TOCE board has generally created a ranked list of about ~15 papers, and the TS program chairs have generally selected about 6 papers from that list and scheduled a few TOCE "sister" sessions. We've done a few tweaks to the timing of this and naming of sessions, but generally it's been the same overall process for several years, and has received positive feedback from authors. Attendance has been reasonable at sessions, and author interest has been high, so we've continued the practice. For context, TOCE receives about 250 submissions per year, many of which are rejections from the SIGCSE TS research track.

Over the years, however, I've talked to many in the community about the symposium's capacity for reviewing and publishing research, a number of challenges have surfaced:

Many are frustrated that the research track is only 6 pages, as it does not allow for complete reporting, sufficient detail for reproducibility or recoverability, limits the ability to sufficiently cite prior work. (Many research papers have published on the these problems and pointed to the symposium's limits as a major cause).

Many are frustrated that the papers are not explicitly labeled as research papers in either the sessions or  proceedings, limiting their perception of peer reviewed work by tenure and promotion committees. Some have reported having colleagues question whether their SIGCSE TS research papers were peer reviewed  in the same way other conference papers are.

Many are frustrated that the review process often does not account for research expertise, leading to reviews that are misinformed and often grounded in individual experiences instead of prior research, theory, and knowledge of research methods.

All of this is survivable — people still submit to the track anyway, and authors can always submit to ACM ICER or TOCE to avoid these problems.

But an idea has emerged, after observing many other ACM communities experiment with PACMs, revise and resubmit models, and other efforts that try to blend conferences and journals. Here's the concept that some have converged on:

Instead of a SIGCSE TS research track, encourage authors to submit to ACM TOCE any time of the year, and to volunteer as a reviewer for ACM TOCE.

Any authors with papers accepted by the October preceding the TS would get a presentation slot at the TS. The conference would become a place to share and discuss the work, but not review it.

Here are the advantages of this:

The TS would not have to manage the research review process, or the nuanced challenges of trying to balance the long-standing tradition of open review in the other tracks with the research track's different needs of accounting for research expertise.

Research submission reviewing would be better consolidated and spread across the year, helping the small but growing community make the most of its limited reviewing capacity, unlike now, where it is fragmented across many venues.

TS program chairs could focus their efforts on curating a great program from accepted TOCE papers, with less divided attention from multiple distinct review processes.

TOCE authors would have a much greater chance to have their work presented than they currently do.

There are of course some disadvantages:

Some authors conference publishing models more than journals, and perceive the reviewing standards to be different. (These perceptions aren't always aligned with reality, but they exist).

Some are in departments that explicitly value conferences over journals.

That's what I've got. I'm happy to chat further about these possibilities if this sounds like something the TS committee would like to explore.

[amyjko]

The board has not replied.

[amyjko]

Steve Wolfman replied with this:

The Steering Committee had a chance to discuss the idea of routing Computing Education Research submissions and reviewing from the SIGCSE TS via TOCE, eventually to be hosted back at the TS.

At a high level, unfortunately, we don’t feel like this would be beneficial for the Technical Symposium. We’re excited about the community the TS has built up encompassing both computing education research and practitioners. This proposal has the potential to create additional barriers between those communities. Additionally, we understand that it might be reasonably common for TOCE authors to be reluctant to attend and present at the TS; given that, this proposal has the potential to reduce participation in the TS for CER-focused authors and participants after they successfully move through the TOCE process.

We’d like to propose some smaller ideas to consider instead:

1. If the sister track has been of added value for TOCE and TOCE authors, perhaps we can consider expanding that to include more papers and sessions. One caveat on that is that the TS received an unusually large number of submissions this year. Responding to that may put extra pressure on our available time slots. However, we think there’s still potential to try out an increase even in TS-24 and, of course, we can discuss this for future years as well.

2. Alternatively, perhaps we can route some submissions in the opposite direction. The TS could identify top submissions in the CER track and work with TOCE to offer to authors some sort of expedited process to submit to TOCE to publish an extended version of their paper. If you’re interested in this for TS-24, we’d want to discuss with the TS Program Chairs rather soon to see if they can fold into their process identifying a reasonable number or percentage of papers from the CER track to select. (Reviewing for TS-24 is well underway, thanks to Steve's belated reply!) Again, we could also discuss for future years, and there are no space/time constraints at the symposium for this proposal.

[amyjko]

This issue is essentially done. The steering committee is interested in keeping the research track as is.