Closed mauriciogtec closed 1 year ago
@mauriciogtec, this is great. I just want to add one more point, all source codes' folders should be under another folder with the same name as the package name.
Here is the modified version:
.
└── packagename
├── LICENSE
├── README.md
├── packagename
│ ├── __init__.py
│ ├── algorithms
│ │ ├── __init__.py
│ │ ├── algorithm1.py
│ │ ├── algorithm2.py
│ │ └── algorithm3.py
│ ├── datasets
│ │ ├── __init__.py
│ │ ├── dataverse_api.py
│ │ ├── metadata_master.yaml
│ │ ├── sampling_models.py
│ │ └── user_api.py
│ ├── evaluation
│ │ ├── __init__.py
│ │ ├── binary_metrics.py
│ │ └── continues_metrics.py
│ └── synthetic_generation
│ ├── __init__.py
│ ├── models.py
│ └── training_api.py
└── setup.py
Distribution should include one module at a time.
maybe have a folder called tools including for example, visualization tools?
probably should add a requirement.txt file too.
This is a summary of desired codebase functionality:
Benchmark dataset (user side): It contains utils to generate new evaluation benchmark datasets. This task requires downloading the core datasets from Dataverse---including their metadata---and combining them with a ``sampling model". Thus, its key components are:
Evaluation Metrics and Tools: Tools for the user to evaluate the performance of a spatial confounding algorithm in a standardized way. It needs to define evaluation metrics. Some examples are:
Training counterfactual/synthetic models. As detailed in the paper, we plan to support various methods to generate synthetic datasets (various types of NNs, forests, and ensembles). This code is not required from the user side, but it must be available for reproducibility purposes. The installation of requires dependencies must be optional. It would also be nice to have an API that users can use to train their own synthetic models.
Spatial confounding algorithms: Here implementations of various algorithms for resolving spatial confounding are implemented using a common interface.
In accordance with the above, here's an example of how the could be reorganized: