Column names in data almost always contain spaces/special characters, are sometimes not unique

[japhir]

Most datasets that I could find contain special characters and spaces, and for a recent dataset turned out not to be unique (resulting in errors when doing further analysis using the tidyverse).

It would be nice if it could apply e.g. janitor::clean_names()

  wes20_1264 <- pangaear::pg_data("10.1594/PANGAEA.917500")[[1]]$data

  colnames(wes20_1264)

  # have to manually rename to: not contain spaces, not contain special characters, be unique
  colnames(wes20_1264) <- c("label",
                            "depth_mbsf",
                            "depth",
                            "age",
                            "d13C",
                            "d18O",
                            "spec",
                            "ref",
                            "masspec",
                            "comment",
                            "frac")

[sckott]

@naupaka @gavinsimpson took over maintaining this pkg, i'll hand it over to them

[naupaka]

Hi @japhir -- thanks for the note. I would be a bit worried about changing the column names in any sort of automated fashion, since it could break a lot of old code from users that might rely on the old names. I think if a user wants to run janitor::clean_names() then they could do so, since they would then be aware of the consequences.

An alternative would be some sort of warning if there are column name issues and perhaps a suggestion/message to the user to consider using janitor::clean_names(). Would that be a reasonable alternative from your perspective?

[japhir]

Yeah I agree that doing this always would probably be too drastic. Ideally we would make sure that future data packages have something like an elaborate description of what each column means and what units it has in the metadata, while the column names still have a letters, periods/underscores and numbers (not as first character) and are guaranteed to be unique. Best would be if they would adhere to some kind of ontology where for example age is always called age and always has the same units (e.g. Ma or ka) and d18O and d13C would always be called that, with the metadata indicating whether they have been adjusted for species-specific vital effects etc.

Perhaps there could be a link between the full column name and the tidied up column name so that if you ever want to get back to what was written originally you can still do so?

I agree that the easiest implementation would be to just write a suggestion message.

In and of itself, having such column names is not too much of a problem because (at least when using the tidyverse) you can wrap them with backtics "`". However, this almost always is very annoying to type and doesn't autocomplete ;-). And this does not work if any of the full column names are duplicates.

[karawoo]

Maybe pangaear could also reexport janitor::clean_names() to make it easier for the user? That way when they see the message they can fix the names right away without having to go and install and load janitor.

[gavinsimpson]

Personally I don't think that's a good idea as it forces a dependency on users who might not otherwise want to do their wrangling using {janitor}. The pages should be as agnostic in that regard as possible.

[robwschlegel]

A way to address the issue of multiple repeat column names for longer variable names, without requiring external dependencies, is to change the behaviour of the internal function pangaer:::pang_GET(), which is called by pg_data(). This is where it creates the tibble from the downloaded data. Currently it calls:

tibble::as_tibble(dat, .name_repair = "minimal")

But it would be a better default behaviour to use:

tibble::as_tibble(dat, .name_repair = "check_unique")
# OR
tibble::as_tibble(dat, .name_repair = "unique")

To ensure backward compatibility the package maintainer could add the argument .name_repair = "minimal" to pg_data(). This would allow users to change this argument as it is passed from pg_data() to pang_GET() to as_tibble().

Currently because tibble::as_tibble(dat, .name_repair = "minimal") limits the character length of column names, if a variable has a long name this tends to cut off the units of measurement. In addition to the issue of columns having multiples of the same name.