fititnt
commented
3 years ago Oh boy, name is complicated.
HXLMeta_Glossary
The EticaAI-Data_HXL-Data-Science-file-formats have an HXLMeta_Glossary to describe concepts (actually, some sort of "ID" to reference on other documents.
This still not the final result, but there is so many concepts on so many programs, that to make some sense or everything I'm trying go put some names. But eventually we could come with some nice ways to mention it.
References
Also, the table EticaAI-Data_HXL-Data-Science-file-formats_References have part of documents I'm just letting there. Some of these may be used later as reference to the actually development.
TODO: software comparison
Maybe at some point will be necessary put on a place some of the software we're testing, but instead of have one spreadsheet for every software, have some by how the terms relate to each other.
I mean: how HXL +text is used on other tools, and how other tools use to be converted back.
Another point is that some of open source tools actually are more powerfully than others and can open several proprietary formats, like the Jamovi (https://www.jamovi.org/), so this table in special could be used by users beyond us to check alternatives.
See:
- EticaAI-Data_HXL-Data-Science-file-formats_HXLMeta_Glossary
One feature of the HXLTabConverter common class #8 (since we're already reading all documentation to see how to make inferences without forcing users to use type hints in all places) actually requires knowing the supposed data types of already HXLated datasets. So, let's break in an separate class [and as much as possible already try to use data structures that could be converted from JSON or something] to create something that actually could make these inferences
The more specific HXL Core hashtags
One advantage of using the hashtag that already is the very own defined on the specification is that the specification for several cases enforce the types. This happens on special for indicators. So, actually, is possible to (at least if is not doing something like brute forcing with the
hxlquickimport) be somewhat sure about what to expect from the data columns.Which accuracy to aim?
In my personal, honest opinion, >90% of the cases is good enough, including making inferences beyond the official documentation (but at this point may need to do some checking on at least a good amount of rows to deduce. But should exist one way that allows users to explicitly enforce (even if it means a more verbose attribute).
Maybe a different approach to tolerate even less accuracy on first try (think like >75%", maybe less) is if is possible to easily import back the exported format (think the .tab from Orange Data Mining, but could be Weka and others) we assume that the data types and data flags (is meta? This can be ignored? Etc) could already be imported back with more data type hints that if exported again would not change.
In other words: for very long spreadsheets, somewhat already optimized to be corrected on an external program. (I think this is much more likely to happen for data flags than data types, in fact we may need to create some way to allow more than one target variable).
How to warn the suggestions outside what already is strictly defined on the HXL Standard
Also, since already do exist the concept of Debug logs, I think when we try to make inferences on the tags that are less than 90% (or maybe we discover that an analysis of 100 (or up to 10.000) the user literally done poor tagging and this is 98% likely to fail on external data mining tools, we still warn the user (This type of feature would be need if trying to brute force with the hxlquickimport, so at least some quick checks could already exist).