datova-kancelaria / nkod-pipeline

Obsahuje export LP-ETL pipeline NKOD pro deployment do k8s
0 stars 1 forks source link

Harvestovanie LKOD Zvolena #11

Open miroslavliska opened 7 months ago

miroslavliska commented 7 months ago

Po harvestovaní LKODu Zvolena vznikne takýto log:

2023-11-09 23:25:21,461 [pool-17594-thread-1] ERROR c.l.e.e.a.v.r.DefaultReportWriter - Task 'https://opendata.zvolen.sk/set/catalog/lkod' failed in 60081 ms com.linkedpipes.etl.executor.api.v1.LpException: Can't download file. at com.linkedpipes.plugin.ehttpgetfile.multiple.DownloadTaskExecutor.accept(DownloadTaskExecutor.java:43) at com.linkedpipes.plugin.ehttpgetfile.multiple.DownloadTaskExecutor.accept(DownloadTaskExecutor.java:12) at com.linkedpipes.etl.executor.api.v1.component.task.TaskConsumerWrap.executeTask(TaskConsumerWrap.java:77) at com.linkedpipes.etl.executor.api.v1.component.task.TaskConsumerWrap.executeTasks(TaskConsumerWrap.java:70) at com.linkedpipes.etl.executor.api.v1.component.task.TaskConsumerWrap.run(TaskConsumerWrap.java:49) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.net.SocketTimeoutException: Connect timed out at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:299) at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:498) at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:603) at java.base/sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:266) at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:380) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:189) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1242) at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1128) at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:175) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1665) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1589) at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529) at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308) at com.linkedpipes.plugin.ehttpgetfile.multiple.Downloader.resolveRedirects(Downloader.java:180) at com.linkedpipes.plugin.ehttpgetfile.multiple.Downloader.connect(Downloader.java:145) at com.linkedpipes.plugin.ehttpgetfile.multiple.Downloader.download(Downloader.java:101) at com.linkedpipes.plugin.ehttpgetfile.multiple.DownloadTaskExecutor.accept(DownloadTaskExecutor.java:41) ... 9 common frames omitted

Mohol by to byť dočasný problém v noci, ale obsah neviem stiahnuť ani teraz, ak sa pripojím z mojej VPN z Nemecka. Teda to vyzerá tak, že tento poskytovateľ dát bráni prístupu zo zahraničia, minimálne z nemeckého cloudu, z ktorého je to potrebné. Kontaktujte, prosím, Zvolen, či je to tak a či to vedia povoliť.

miroslavliska commented 7 months ago

Zatváram tento ticket. Harvestovanie Zvolena ide. To že nie sú datasety ešte vyhľadateľné je nahodené v iných ticketoch. Tu je potvrdenie: image

miroslavliska commented 6 months ago

V katalogu Zvolena sa nepoužíva správne URI formátu. Formát v URI musí byť kapitalizovaný. pr.: http://publications.europa.eu/resource/authority/file-type/JSON

miroslavliska commented 2 months ago

Chyby objavené v LKOD Zvolena:

miroslavliska commented 1 month ago

Portál nesprávne zobrazuje informáciu o chýbajúcej licencii, ktorá tam ale je. Treba to preveriť.