Open Data Portal

[ValterAro]

Problems with opendataportal ("Avaandmed") in the bürokratt analytics module

Related issues

3 #24 #21 #59

Test evironment:

https://admin.dev.buerokratt.ee/analytics/reports using Google Chrome Version 116.0.5845.96 (Official Build) (64-bit) for Ubuntu

Description:

1. API Configuration Issue (Fixed 19.09.2023):

   • When configuring the api key, api key id, and the org id using the "Seadista teabevärava juurdepääs" button, the website throws an error: "Viga ühenduse loomisel. API võti või API võtme ID ei olnud õiged" even if the credentials are correct.
   • On navigating to another menu section and then returning, the generated API key appears at the bottom even when the initial data was incorrect.

2. "Loo uus andmestik teabeväravas button" runtime error (Fixed):

   • The "Loo uus andmestik teabeväravas" button opens a window, but triggers a runtime error. Closing the error also closes the window. The window only displays the title "Loo uus andmestik teabeväravas".

3. CSV Download Issue(Fixed):

   • The "Lae alla CSV" button triggers a runtime error and does not download a csv.

Details:

2. "Loo uus andmestik teabeväravas" button runtime error:

• Network: 2 times the following request:

  Request URL: https://ruuter.dev.buerokratt.ee/an/odp/values?lang=et
  Request Method: GET
  Status Code: 300
  Remote Address: 171.22.247.156:443
  Referrer Policy: strict-origin-when-cross-origin

• GUI: Uncaught runtime errors:

  ERROR
  Request failed with status code 300
  AxiosError: Request failed with status code 300
  at settle (https://admin.dev.buerokratt.ee/analytics/static/js/bundle.js:270273:12)
  at XMLHttpRequest.onloadend (https://admin.dev.buerokratt.ee/analytics/static/js/bundle.js:268985:66)

• Console:

  i18next::translator: missingKey et common global.save global.save
  i18next.js:25 i18next::translator: missingKey et common global.save global.save
  index.tsx:94 Uncaught (in promise) AxiosError ...

  3. CSV Download Issue:

• Network:

• Request 1:

  Request URL: https://ruuter.dev.buerokratt.ee/an/odp/download
  Request Method: OPTIONS
  Status Code: 200
  Remote Address: 171.22.247.156:443
  Referrer Policy: strict-origin-when-cross-origin

• Request 2:

  Request URL: https://ruuter.dev.buerokratt.ee/an/odp/download
  Request Method: POST
  Status Code: 502
  Referrer Policy: strict-origin-when-cross-origin

• GUI: Uncaught runtime errors:

  ERROR
  Network Error
  AxiosError: Network Error
  at XMLHttpRequest.handleError (https://admin.dev.buerokratt.ee/analytics/static/js/bundle.js:269034:14)

• Console:

  Access to XMLHttpRequest at 'https://ruuter.dev.buerokratt.ee/an/odp/download' from origin 'https://admin.dev.buerokratt.ee/' has been blocked by CORS policy ...
  xhr.js:251 POST https://ruuter.dev.buerokratt.ee/an/odp/download net::ERR_FAILED 502
  dispatchXhrRequest @ xhr.js:251
  ...
  WebSocketClient.js:13 WebSocket connection to 'wss://admin.dev.buerokratt.ee:3010/ws' failed:

[ValterAro]

---

Tested Build: Analytics-Module-b6ca0ba6ed489a1aada09caa0efff7acb09e471c - (this commit was on March 30) the last version, which was working and where Ahmed fixed the latest bugs as of 29.08.2023

Environment:

• Local setup on my computer with help from Ahmed.
• Using the FortiClient VPN.
• Tested (http://localhost:3001/analytics/reports) on Google Chrome Version 116.0.5845.96 (Official Build) (64-bit)
• Ubuntu 20.04

Findings:

1. Issues faced in this bug report were not present.
2. Successfully retrieved the CSV.
3. Entered the metadata successfully using the "Loo uus andmestik teabeväravas" button.
4. Issue: Unable to publish the test data file on avaandmedtest.eesti.ee. (Fixed 19.09.2023)
   • Metadata was visible on avaandmedtest.eesti.ee, but the attached data was not present.
5. Occasionally, the request does not go through.
6. Add a "Andmed kuni" field in the metadata entry form. This field should indicate the date up to which the data is collected.

[ValterAro]

• [ ] There is an issue when pressing the "Muuda" button then the original metadata shows mock data instead of some of the metadata that was entered and the rest of the data is different from the original. Tested this also on the https://admin.dev.buerokratt.ee/analytics/reports site in addition to the local testing and saw the same problem.

In order to recreate: 1) Go to https://admin.dev.buerokratt.ee/et/dev-auth and login 2) Go to https://admin.dev.buerokratt.ee/analytics/reports and press "Seadista teabevärava juurdepääs" button and use the API key and API key ID with the organization ID. 3) After the API credentials are verified, use the "Loo uus andmestik teabeväravas" and enter the metadata 4) After saving the metadata the data is published in open data portal and you can see it in the "Loodud andmestikud" table 5) When trying to change this data it shows the mock data instead of the original data that was entered

[ValterAro]

• [x] Additionally the createDataset should use the url for publishing data as an organization and not as a user, which is the current solution. Meaning that the data is published under my username instead of the organization

-Code in the file .../DSL/Ruuter/POST/odp/dataset.yml

createDataset:
  call: http.post
  args:
    url: https://avaandmedtest.eesti.ee/api/users/me/datasets # replace with organization url that contains org id
    body:

More information https://avaandmedtest.eesti.ee/api/dataset-docs/#/

[ValterAro]

As written in issue #3 acceptance criteria:

• [ ] Possibility to set interval for automatic data uploads to Open Data Portal (daily, weekly, monthly)
• [ ] Possibility to send data to Open Data Portal manually

In both cases there is no file added in the avaandmeddev.eesti.ee website although you can see the metadata there. To clarify the second issue. Publishing data manually, by selecting "Uuenemissagedus" - "Ei uuenda" in the metadata insertion and selecting save it fails and shows a text "Andmestiku salvestamine ebaõnnestus"

[1AhmedYasser]

• [ ] There is an issue when pressing the "Muuda" button then the original metadata shows mock data instead of some of the metadata that was entered and the rest of the data is different from the original. Tested this also on the https://admin.dev.buerokratt.ee/analytics/reports site in addition to the local testing and saw the same problem.

In order to recreate:

1. Go to https://admin.dev.buerokratt.ee/et/dev-auth and login
2. Go to https://admin.dev.buerokratt.ee/analytics/reports and press "Seadista teabevärava juurdepääs" button and use the API key and API key ID with the organization ID.
3. After the API credentials are verified, use the "Loo uus andmestik teabeväravas" and enter the metadata
4. After saving the metadata the data is published in open data portal and you can see it in the "Loodud andmestikud" table
5. When trying to change this data it shows the mock data instead of the original data that was entered

@turnerrainer Currently when fetching a dataset we use this

get_dataset_metadata:
  call: http.get
  args:
    url: "[#ANALYTICS_OPEN_DATA_SERVICE]/api/organizations/my-organizations/[#OPEN_DATA_PORTAL_ORGANIZATION_ID]/datasets/${incoming.params.datasetId}"
  headers:
      Authorization: ${'Bearer ' + apiToken.response.body.data.accessToken}  
  result: dataset

Currently ruuter does not understand parameters or body parameters passed to the url unless its defined in constants.ini, we need to modify it to allow for url to accept parameters as a part of the url

[1AhmedYasser]

As written in issue #3 acceptance criteria:

• [ ] Possibility to set interval for automatic data uploads to Open Data Portal (daily, weekly, monthly)
• [ ] Possibility to send data to Open Data Portal manually

In both cases there is no file added in the avaandmeddev.eesti.ee website although you can see the metadata there. To clarify the second issue. Publishing data manually, by selecting "Uuenemissagedus" - "Ei uuenda" in the metadata insertion and selecting save it fails and shows a text "Andmestiku salvestamine ebaõnnestus"

Waiting for Ruuter Changes to read body or parameters inside the url

[ValterAro]

I was able to upload a file to avaandmeddev.eesti.ee through postman using the following request

curl --location 'https://avaandmeddev.eesti.ee/api/organizations/my-organizations/f35dee7a-2970-442e-8e3a-9cf852c060e5/datasets/2aca1eb9-d3e6-4185-9e75-cf1a6b7f14e4/upload' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyRW1haWwiOiJ2YWx0ZXJhcm9AZ21haWwuY29tIiwic3ViIjoiMzZhZGMwODAtMjcwZS00MDMxLWE3MTAtNDMyZTM0NjA2NWI4IiwiYXBpS2V5SWQiOiI4NmQxNDFjYy1hYzQ2LTQ2YmMtODI5Yy1mMmQ2YWNiNDAzZmMiLCJpYXQiOjE2OTY0MDI3ODUsImV4cCI6MTY5NjQwMzY4NX0.sa96PngI1JJKXdjQnWglEE9Z2tGz0PWh36nKz-H6B9o' \
--form 'files=@"/home/valtera/Downloads/metrics (12).csv"' \
--form 'delimiter=","' \
--form 'headerIndex="1"'

The Ruuter needs added functionality that enables it to upload files (the required format seems to be binary as per the website requests) Also, for the dataset to work it needs to be published or discarded. For example using the following request:

curl --location --request PUT 'https://avaandmeddev.eesti.ee/api/organizations/my-organizations/f35dee7a-2970-442e-8e3a-9cf852c060e5/datasets/2aca1eb9-d3e6-4185-9e75-cf1a6b7f14e4/publish' \

[ValterAro]

Ahmed will create a PR and this goes into acceptance testing. Discussion will continue under issue #3 when necessary.

[1AhmedYasser]

@turnerrainer @rasmusei @PaulaMerle Fixed ODP with @ValterAro , now you can create, view , edit and delete datasets and the datasets are linked to the organization and you can also upload and publish them.

@turnerrainer Now This still needs ruuter changes to add the cronjob ability to run scheduled uploads for datasets as mentioned here Possibility to set interval for automatic data uploads to Open Data Portal (daily, weekly, monthly) in https://github.com/buerokratt/Analytics-Module/issues/3 and it will be addressed in development here https://github.com/buerokratt/Analytics-Module/issues/121

PR is up https://github.com/buerokratt/Analytics-Module/pull/118 Thank you,

[turnerrainer]

Cronjob by #122, consider this task as done.

[ValterAro]

Upload and publish is not working for the odp. http://localhost:3001/analytics/reports. I am able to create a new dataset, but when I choose "update interval" as "never" then the file is not uploaded and dataset is not published. The expected behavior is that, when "never" is selected during the dataset creation then a file is uploaded to the current dataset and it is also published.

[ValterAro]

Ruuter was updated by Varmo for the server and the issue with odp upload and dataset publishing was fixed.

[ValterAro]

[BUG] Short summary: Error when trying to send a dataset to avaandmedtest.eesti.ee (testing admin.test.buerokratt.ee). How to reproduce: Go to https://admin.test.buerokratt.ee/analytics/reports and try to create a new dataset for odp. When trying to save it, then there is a alert saying it failed. Pic:

[ValterAro]

[BUG] Short summary: Download csv does not work in https://admin.test.buerokratt.ee/analytics/reports . Also mentioned here https://github.com/buerokratt/Analytics-Module/issues/216#issue-2289274807 . The other csv downloads in analytics module work.

Pic: