Files not supported by the ODE

[romicolman]

Problem description

When you upload a file that is not supported by the ODE, you get the following message on the ODE screen:

Steps to reproduce it

• Download the DMG file from OpenRefine.
• Open the ODE and click ADD (left menu)
• Upload the file

Suggested solution

Whenever you upload a file not supported by the ODE you should get this message:

Your file is not supported by this tool

• I would not use the name of the tool, to avoid updating the message if the name changes.
• ODE should not display the file on the panel on the left

[roll]

@romicolman @pdelboca The problem here is that the mainstream approach of using the application is not "uploading" files but opening local folders that already have a lot of files. It's a UX question: what is better to hide not supported files under the flag or use another approach. In-general, as a user I might be very confused if in a folder I see 50 files and in ODE showing the same folder I see 10 files (where are my files?). We again can learn from what VSCode etc does in this case

[guergana]

@romicolman

Hello, I think I need a more specific instruction here. How/Where is the message: Your file is not supported by this tool displayed? in which panel? Is it a popup? Can I see a visual representation of what you expect as the result?

[sapetti9]

@guergana you'll find a list of supported data formats by the Frictionless-py here: https://framework.frictionlessdata.io/ under Data Formats

[romicolman]

@guergana you can add a pop-up window to inform the user that the file is not supported by the tool.

Message: Your file is not supported by this tool

Supported data formats: CSV, XLS, JSON, SQL

Let me know if you have further questions

[guergana]

@guergana you can add a pop-up window to inform the user that the file is not supported by the tool.

Message: Your file is not supported by this tool

Supported data formats: CSV, XLS, JSON, SQL

Let me know if you have further questions

Hi, I do have questions @romicolman I tried importing the files I had at hand and they are the ones from the repository itself. They are code so they are all text files and all of them are opened by the editor. One of these files is a yaml file... In the link to the frictionless api docs provided by Sara, it says CSV, XLS, JSON, SQL, and others. I think we shouldn't have this ambiguity, which are the other formats? Is yaml included in the supported types? yaml is a subset of json. Also I see the app already supports maps in the form of .geojson and images... I need the full list of extensions. 🙏

Another thing is that the add button lets the user adds several files.. What do we do in the case the user tries to add a file with a valid extension and one with an unsupported extension? Do we send the message and dont import any of the files? or do we import the files with the supported extension?

[roll]

@romicolman @guergana Take into account that the app is intended to be used on local directories users open. It means that there will be always 0-N files that can be opened by a data viewer but they still need to be present in the file browser (so user is not confused -- where are my files) and they need to be available for metadata editing and adding to a data package.

For example, consider a simple projects having:

• table.csv
• paper.pdf

There is no view available for paper.pdf but as as USER a still needs to be able to describe and package this file.

[roll]

@sapetti9 There are also formats supported by ODE which is not tabular formats that supported by frictionless-py

[roll]

Files support:

• LIST: all files (file browser)
• META: all files (metadata editor)
• VIEW: subset of files (contents representation)

[romicolman]

@romicolman @pdelboca The problem here is that the mainstream approach of using the application is not "uploading" files but opening local folders that already have a lot of files. It's a UX question: what is better to hide not supported files under the flag or use another approach. In-general, as a user I might be very confused if in a folder I see 50 files and in ODE showing the same folder I see 10 files (where are my files?). We again can learn from what VSCode etc does in this case

• Yes, I understand this, but as users Open gives the idea of opening a file that already exists on the application. I checked what OCCRP has for Aleph in their investigations section and they use the same term (Upload). Another option is to replace Upload with Browse (saw this on another open source tool), but it is less transparent for users.

• Regarding files not supported by the ODE. It's not a good idea to show the user that the file is not supported and then added it to the list of files on the left menu. First scenario: let's say we decide to hide the non-supported file. Will the non-supported file be exported among others when the user downloads data? If that's the case.. this is problematic. So, my second scenario (the best one, but I don't know if it's possible): reject the non-supported file (do not store it on the ODE) and show the user a message ("Your file is not supported by this tool"). Adding @guergana here for comments

[romicolman]

@guergana you can add a pop-up window to inform the user that the file is not supported by the tool. Message: Your file is not supported by this tool Supported data formats: CSV, XLS, JSON, SQL Let me know if you have further questions

Hi, I do have questions @romicolman I tried importing the files I had at hand and they are the ones from the repository itself. They are code so they are all text files and all of them are opened by the editor. One of these files is a yaml file... In the link to the frictionless api docs provided by Sara, it says CSV, XLS, JSON, SQL, and others. I think we shouldn't have this ambiguity, which are the other formats? Is yaml included in the supported types? yaml is a subset of json. Also I see the app already supports maps in the form of .geojson and images... I need the full list of extensions. 🙏

Another thing is that the add button lets the user adds several files.. What do we do in the case the user tries to add a file with a valid extension and one with an unsupported extension? Do we send the message and dont import any of the files? or do we import the files with the supported extension?

Answers below

1) Full list of extensions: @roll I read your comments. Can you provide the full list of extensions the ODE should support to work properly? One comment regarding this. For the stable release we will concentrate on describing and validating tabular data****, (no images or articles, for instance). Can you list everything the ODE should consider? These are the formats I see across comments:

• CSV, XLS, JSON, SQL
• LIST: all files (file browser)
• META: all files (metadata editor)
• VIEW: subset of files (contents representation)

Do we need to include paper.pdf? This is not tabular data but I wonder if you mention it for some specific reason. What about ?

2)" Another thing is that the add button lets the user adds several files.. What do we do in the case the user tries to add a file with a valid extension and one with an unsupported extension? Do we send the message and dont import any of the files? or do we import the files with the supported extension?"

• @guergana: Upload all files and for the non-supported file display the following message to the user: "[ NAME OF THE FILE] could not be uploaded since it is not supported by this tool" Way to show it: pop-up message.

[guergana]

@romicolman moving back to Backlog because the solution is to add all the files, right? and it's not part of the sprint.

[romicolman]

Yes, we will add all files. Let's discuss this in the planning meeting today.

[guergana]

Yes, we will add all files. Let's discuss this in the planning meeting today.

And then the message displayed in the viewer stays the same as now or do we update it ? In case we update it which would be the new message? @romicolman

The one now is This file type does not have a supported data view

[romicolman]

Please, add this message: Preview is not available for this file format Why? Because the ODE in reality supports the file, the issue is that since it is not tabular data, it cannot be shown.

Context

Full list of formats supported by the Frictionless Framework

[roll]

Note, that the list from Frictionless Framework includes not only tabular formats, and a format needs to be enabled in Open Data Editor separately - https://github.com/okfn/opendataeditor/blob/main/pyproject.toml#L5 (currently, we support CSV/Excel/JSON/Parquet)