documentation issues

[MangoIV]

When trying to use this tutorial, I encountered several issues:

• it is unclear how the python code of the user is being called by tira, i.e. if tira doesn't find the code it just say "dataset not found, are you sure it's there?" which is like the most unhelpful error message, ever.
• it is unclear which document formats are expected, e.g. this works for python files, but not for ipynb
• it is unclear where the dependencies come from, i.e. there are things imported in the example file but it is impossible to troubleshoot errors with this, because, it neither seems to adhere to conventions, nor is it documented, where the source code for this can be actually found. It is insufficient to have people rely on some opaque docker container.
• it is unclear whether in the actual tira, the docker contain will have web access, in the tutorial you are actually using web access to be able to use your tools to pull the assets from the same repo that the files are already in there, however, in some places you say, that it is important that files are available locally.
• JsonlDocs does neither accept a file, nor a path, nor a path string, what does it accept then?

[mam10eks]

Dear @MangoIV,

Thank you for bringing up the issue. The dependency used here is ir_datasets. Documentation in research software is always a bit hard because things change fast.

We used this tutorial now for the very first time with bachelor and master students, and I think the feedback, questions, and problems that we saw will help us to improve this tutorial and its documentation further. We tried to encourage questions and discussions, and I think this worked quite well for the first iteration of the IR lab.

For future iterations of the IR lab, I think I learned that we should already provide the complete ir_datasets integration, so that only specialized needs have to adopt the ir_datasets registrations and most use cases are already covered by the template that we give. In that scenario, the goal for the first milestone (dataset creation) is to implement a preprocessing script/program that outputs a file, which should be much simpler.

For your questions:

• The "dataset not found" error is from ir_datasets when the class is not registered. We can tackle this problem in the future by already providing the complete ir_datasets integration as described above.
• For the document formats: python does, by default, not load classes defined in Jupyter notebooks (would also be solved when we provide the ir_datasets integration in advance).
• For the dependency: we only use ir_datasets in the tutorial, we will make this more explicit (but this would also be solved by my suggestion above).
• For the internet connection: data imports can access the internet in TIRA. The subsequent software executions (which will be milestone 2 in the IR lab) don't have access to the internet.
• Indeed, there is not much documentation for JsonlDocs, but we have provided an example on how to load data from the internet and an example on how to load data from the local file system, I hope this covers all the use cases that we encounter in the IR lab.

What do you think about the idea that I described above? Should still allow all possibilities, but should substantially simplify everything, or?

Best regards,

Maik

[MangoIV]

Hi, thank you for your comprehensive answer. This is a great help.

I don't think anything needs to be simplified. I still feel like it would be good to leave a comment of intended use of certain things, ie I couldn't immediately figure out that for local file we want that packagesmth class and then searched for something that sounded right encountering new issues (there's a RelPath class which takes a "streamer" which is documented nowhere, so what class is expected there?)

So as a conclusion perhaps if the software is used by some third party like in this case, it would be good to leave a docstring or a type signature (or annotation in Python?) which typically don't tend to get stale.

This is perhaps also true for the ir_datasets dep.

Please mind that there are several people who encountered this issue, so I'm really not the only one.

Thanks again for the comprehensive and fast answer.

[mam10eks]

I now improved the documentation, and we also have created a dedicated example for the dataset used in the IR lab: https://github.com/mam10eks/ir-lab-sose-2023/tree/main/milestone-01

I think in future iterations of the IR lab, we will directly give a dedicated template as the one linked above, because this than allows to put more focus on the interesting parts and not technical details.

Thanks again for bringing up the issue,

Best regards,

Maik