okfn / docker-ckan

Docker images and Docker Compose setup for CKAN [Not Maintained]
GNU Affero General Public License v3.0
81 stars 91 forks source link

Datapusher not working on my local enviroment #73

Open JeffreyVRiemsdijk opened 3 years ago

JeffreyVRiemsdijk commented 3 years ago

My environment is localhost with nothing changed except only these two settings in my hosts file on windows to try and find out if it worked but sadly enough both solutions do not work:

127.0.0.1 ckan
127.0.0.1 datapusher

After uploading a dataset i get this:

ckan-dev_1    | 2021-06-10 17:12:11,273 DEBUG [ckanext.datapusher.plugin] Submitting resource 1caafc34-ef92-4797-aaf2-d18693f9cfc4 to DataPusher
solr          | 2277933 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=name:"695790ff-759d-4f73-8eb4-4552c526b64f"+OR+id:"695790ff-759d-4f73-8eb4-4552c526b64f"&fq=site_id:"default"&rows=1&wt=json} hits=1 status=0 QTime=2
datapusher    | [pid: 11|app: 0|req: 1/1] 192.168.96.6 () {36 vars in 435 bytes} [Thu Jun 10 17:12:11 2021] POST /job => generated 700 bytes in 101 msecs (HTTP/1.1 200) 2 headers in 72 bytes (1 switches on core 0)
solr          | 2278253 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
solr          | 2278253 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7aa047f1    
solr          | 2278601 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
solr          |         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr/server/solr/ckan/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@336e3983; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_11,generation=37}
solr          |         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solr/server/solr/ckan/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@336e3983; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_12,generation=38}
solr          | 2278601 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.SolrDeletionPolicy newest commit generation = 38
solr          | 2278608 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f016b84[ckan] main]
solr          | 2278609 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@6f016b84[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_q(6.2.1):C10/1:delGen=1) Uninverting(_s(6.2.1):C1) Uninverting(_t(6.2.1):C1) Uninverting(_u(6.2.1):C1) Uninverting(_v(6.2.1):C1) Uninverting(_y(6.2.1):C1) Uninverting(_11(6.2.1):C1)))}
solr          | 2278611 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.QuerySenderListener QuerySenderListener done.
solr          | 2278611 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.DirectUpdateHandler2 end_commit_flush
solr          | 2278611 INFO  (searcherExecutor-7-thread-1-processing-x:ckan) [   x:ckan] o.a.s.c.SolrCore [ckan] Registered new searcher Searcher@6f016b84[ckan] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_q(6.2.1):C10/1:delGen=1) Uninverting(_s(6.2.1):C1) Uninverting(_t(6.2.1):C1) Uninverting(_u(6.2.1):C1) Uninverting(_v(6.2.1):C1) Uninverting(_y(6.2.1):C1) Uninverting(_11(6.2.1):C1)))}
solr          | 2278623 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.u.p.LogUpdateProcessorFactory [ckan]  webapp=/solr path=/update params={commit=true}{add=[ad8d186f2cbca8adf1c72ec67e71deb1 (1702200744687435776)],commit=} 0 398
ckan-dev_1    | 2021-06-10 17:12:12,107 INFO  [ckan.config.middleware.flask_app]  /dataset/done/resource/new render time 1.420 seconds
solr          | 2278830 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=name:"done"+OR+id:"done"&fq=site_id:"default"&rows=1&wt=json} hits=1 status=0 QTime=4
solr          | 2279326 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=50&df=text&fl=id+validated_data_dict&fq=%2Bowner_org:"6d6295db-36ff-40ba-a795-76ddd8106bc4"&fq=%2Bsite_id:"default"&fq=%2Bstate:active&sort=score+desc,+metadata_modified+desc&facet.mincount=1&rows=0&facet=false&wt=json} hits=11 status=0 QTime=3
solr          | 2279349 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=-1&facet.field=groups&facet.field=owner_org&fl=groups&fq=%2Bsite_id:"default"&fq=%2Bstate:active&facet.mincount=1&rows=2&facet=true&wt=json} hits=11 status=0 QTime=4
solr          | 2279824 INFO  (qtp1989972246-13) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=50&df=text&fl=id+validated_data_dict&fq=%2Bowner_org:"6d6295db-36ff-40ba-a795-76ddd8106bc4"&fq=%2Bsite_id:"default"&fq=%2Bstate:active&sort=score+desc,+metadata_modified+desc&facet.mincount=1&rows=0&facet=false&wt=json} hits=11 status=0 QTime=1
solr          | 2279845 INFO  (qtp1989972246-20) [   x:ckan] o.a.s.c.S.Request [ckan]  webapp=/solr path=/select params={q=*:*&facet.limit=-1&facet.field=groups&facet.field=owner_org&fl=groups&fq=%2Bsite_id:"default"&fq=%2Bstate:active&facet.mincount=1&rows=2&facet=true&wt=json} hits=11 status=0 QTime=1
ckan-dev_1    | 2021-06-10 17:12:13,468 INFO  [ckan.config.middleware.flask_app]  /dataset/done render time 1.321 seconds

when i go to manage the resource and then datastore it says:
Error: Process completed but unable to post to result_url

When i click the upload to datastore it says Error: Process completed but unable to post to result_url

And below that it display pending: 9 minutes

Aswell as when i try to preview the resource it keeps saying not available.

ondics commented 3 years ago

Same problem here.

Standard setup. after docker-compose up, uploading a csv file anf starting the datapusher, the datapusher returns with

Error: Process completed but unable to post to result_url 

My .env (excerpt):

# Other services connections
CKAN_SOLR_URL=http://solr:8983/solr/ckan
CKAN_REDIS_URL=redis://redis:6379/1
CKAN_DATAPUSHER_URL=http://datapusher:8800
CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

Where to debug? What's the problem? Has it worked before?

ondics commented 3 years ago

ok, solved:

The .env variables CKAN_SITE_URL and CKAN__DATAPUSHER__CALLBACK_URL_BASE must be identical.

If tey are not identical, the error is

Error: Process completed but unable to post to result_url