Prep data for paper

[jradavenport]

Need to have a unified, value added file format for each conference in the paper. Each data_final.csv file should contain:

• Timestamp
• Talk ID
• Session chair gender
• Session subject area
• Speaker gender
• Question(s) gender

Files should be unique'd (no repeat entries)

We should also keep the raw data file for each conference, which contains (for 225 and 227) the comments field.

Checklist for processing these files:

• [x] AAS 223
• [x] AAS 225
• [ ] AAS 227
• [ ] AAS 229
• [ ] AAS 231
• [x] Cool Stars 18
• [x] Cool Stars 19
• [ ] Cool Stars 20

[PaulDeStefano]

I think I can contribute here, too.

Do you just want the data_final.csv files, or do you want a program that produces data_final.csv from raw data in all three conferences?

[jradavenport]

Just the files. In future surveys we will continue to approach the ideal file format from the beginning :)

---

Sent from my smartphone

On Jan 26, 2016, at 10:08 PM, Paul DeStefano notifications@github.com wrote:

I think I can contribute here, too.

Do you just want the data_final.csv files, or do you want a program that produces data_final.csv from raw data in all three conferences?

— Reply to this email directly or view it on GitHub.

[PaulDeStefano]

Okay, I plan to look into this tomorrow. I could be really easy; fingers crossed.

[PaulDeStefano]

Couple questions:

• Header line, yes or no?
• Header line format: lower-case only, capitalize first letter, spaces okay, replace space with underscore, etc.?
• Time-stamp format: 'YYYY-MM-DD HH:MM:SS' (as given in raw file)? If missing HHMMSS, fill zero?
• Session subject: number codes or do you want these converted into descriptions, somehow. Looks like there is some data in AAS227 that could be used to do this, but I'm not sure and I don't see similar data for the other meetings. (Aha. I see categories.csv in parent directory. session number -> category number -> category name? Can we still get session categories for 223?)
• What to do if there is a name in the subject field?

[PaulDeStefano]

Well, it was trivial to reorder fields and apply some inflexible formatting rules. I did that for all three AAS meetings, but now I have more questions.

• AAS223: field 4 is the gender of session chair, right? For that meeting, speaker gender wasn't recorded?
• For missing data, such as speaker gender in AAS223, what null value do you want? "null"?
• In order to do category names, I need mapping from session number to category. Sorry, but I forgot if the session number has that encoded; just remind me. Or point me to mapping data.

[jradavenport]

Hey @PaulDeStefano, I'm back online after a weekend scuba diving (poor me), here's some thoughts:

• Don't worry about adding category names or numbers to each data entry. That is being done for AAS 223 and 225 in Issue #23 by @hmchristenson and @jneilcottle.
• header line: yes (is my vote). lowercase or CamelCase. single words if possible (underscore OK, but not desired)
• Timestamps: yes. They should be in the YYYY-MM-DD HH:MM:SS format. AAS223 and 227 have them in the data.csv files, AAS 225 has it in the raw.csv, so you'll need to rip it from there.
• I want to drop the "epoch" column, the first giant string of numbers. It is useless for us at present
• column order should be: [Timestamp, TalkID, ChairGender, SpeakerGender, QuestionGenders] and then we'll add SessionCategory as a final column in #23
• AAS 223 did not have session chairs in the survey, so we had to add them ex post facto for the first paper. I am attaching the file we created manually so you can fold it in to the master datafile. It should be included in the repository as well for completeness I think

AAS 223 Session Chair Gender File

[stephtdouglas]

Additional work - categorizing the sessions at each AAS (for #19 / #27):

• [ ] AAS 223 (use https://files.aas.org/aas223/AAS_223_Abstracts.pdf)
• [x] AAS 225
• [x] AAS 227
• [ ] AAS 229 (https://aas.org/files/aas229-abstracts-pdf.pdf)
• [ ] AAS 231 (https://aas.org/files/aas_231_abstracts.pdf)