search

olivere / elastic

Deprecated: Use the official Elasticsearch client for Go at https://github.com/elastic/go-elasticsearch
https://olivere.github.io/elastic/
MIT License
7.43k stars 1.15k forks source link

Deploy Elasticsearch in google cloud engine error #1162

Closed JeffreyLiuUSC closed 5 years ago

JeffreyLiuUSC commented 5 years ago

Please use the following questions as a guideline to help me answer your issue/question without further inquiry. Thank you.

Which version of Elastic are you using?

[ ] elastic.v7 (for Elasticsearch 7.x) [ ] elastic.v6 (for Elasticsearch 6.x) [ ] elastic.v5 (for Elasticsearch 5.x) [x] elastic.v3 (for Elasticsearch 2.x) [ ] elastic.v2 (for Elasticsearch 1.x)

Please describe the expected behavior

Simple backend code for handling search and post, worked well on localhost.

Please describe the actual behavior

On google cloud platform VM instance, keep showing error when type "gcloud app deploy" that

Any steps to reproduce the behavior?

Phase 0:

  1. Everything works well on localhost

Phase 1:

  1. initially on google cloud platform I set VM instance to be at zone : us-central, but when first deploy elastic I choose zone: us-west. don't know if this has impact on later errors.
  2. "gcloud app deploy" successfully on google in the first time with "client, err := elastic.NewClient(elastic.SetURL(ES_URL), elastic.SetSniff(false))".
  3. However I found that my code is not working when I use postman to GET or POST, receiving [] when POST, and receiving "Bad 400" when GET.
  4. I thought might be something wrong but don't know where, so I re-create the google platform and its project and the VMinstance(us-central) and re-deploy main.go in matching zone(us-central)
  5. Still has the same issue with GET/POST

FYI: cloud log: 2019-08-08 04:43:22 default[20190807t214037] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.F4tt/main.go:40 +0x42d 2019-08-08 04:43:22 default[20190807t214037] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.F4tt/main.go:40 +0x42d 2019-08-08 05:00:19 default[20190807t215732] started-service 2019-08-08 05:00:20 default[20190807t215732] started-service 2019-08-08 05:06:00 default[20190807t215732] Received one post request 2019-08-08 05:06:01 default[20190807t215732] Post is saved to Index: test111111111 2019-08-08 05:06:18 default[20190807t215732] Received one post request 2019-08-08 05:06:18 default[20190807t215732] Post is saved to Index: test111asdasd333111111 2019-08-08 05:08:09 default[20190807t215732] "GET /" 404 2019-08-08 05:08:09 default[20190807t215732] "GET /favicon.ico" 404 2019-08-08 05:08:17 default[20190807t215732] "GET /" 404 2019-08-08 05:08:40 default[20190807t215732] Received one request for searchSearch received: 37.000000 -112.000000 200km 2019-08-08 05:08:41 default[20190807t215732] Query took 178 milliseconds 2019-08-08 05:08:41 default[20190807t215732] Found a total of 0 post 2019-08-08 05:09:17 default[20190807t215732] Received one post request 2019-08-08 05:09:18 default[20190807t215732] Post is saved to Index: test111asdasd333111111 2019-08-08 05:09:29 default[20190807t215732] Received one request for searchSearch received: 37.000000 -112.000000 200km 2019-08-08 05:09:29 default[20190807t215732] Query took 12 milliseconds 2019-08-08 05:09:30 default[20190807t215732] Found a total of 0 post 2019-08-08 05:09:43 default[20190807t215732] Received one post request 2019-08-08 05:09:44 default[20190807t215732] Post is saved to Index: test11asdxxxxxx1asdasd333111111 2019-08-08 05:11:18 default[20190807t215732] panic: Head http://104.154.101.215:9200/around: dial tcp 104.154.101.215:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.aEIn/main.go:45 +0x452 2019-08-08 05:13:52 default[20190807t215732] started-service 2019-08-08 05:14:26 default[20190807t215732] started-service 2019-08-08 05:16:40 default[20190807t221330] panic: Head http://104.154.101.215:9200/around: dial tcp 104.154.101.215:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.4NiT/main.go:45 +0x452 2019-08-08 05:20:17 default[20190807t215732] panic: Head http://104.154.101.215:9200/around: dial tcp 104.154.101.215:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.aEIn/main.go:45 +0x452 2019-08-08 05:24:36 default[20190807t215732] panic: Head http://104.154.101.215:9200/around: dial tcp 104.154.101.215:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.aEIn/main.go:45 +0x452 2019-08-08 05:30:48 default[20190807t222749] panic: Head http://104.154.101.215:9300/around: dial tcp 104.154.101.215:9300: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.niQm/main.go:45 +0x452 2019-08-08 05:30:50 default[20190807t222749] panic: Head http://104.154.101.215:9300/around: dial tcp 104.154.101.215:9300: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.niQm/main.go:45 +0x452 2019-08-08 05:49:16 default[20190807t224659] started-service 2019-08-08 05:49:24 default[20190807t224659] started-service 2019-08-08 05:52:35 default[20190807t224659] started-service 2019-08-08 05:55:06 default[20190807t224659] started-service 2019-08-08 05:56:22 default[20190807t224659] started-service 2019-08-08 05:56:53 default[20190807t224659] Received one post request 2019-08-08 05:56:53 default[20190807t224659] Post is saved to Index: test11asdxxxxxx1asdasd333111111 2019-08-08 05:57:39 default[20190807t224659] Received one post request 2019-08-08 05:57:39 default[20190807t224659] Post is saved to Index: test11asdxxxxxx1asdasd333111111 2019-08-08 05:57:46 default[20190807t224659] Received one post request 2019-08-08 05:57:46 default[20190807t224659] Post is saved to Index: test11asdxxxxxx1asdasd333111111 2019-08-08 05:58:47 default[20190807t224659] Received one post request 2019-08-08 05:58:47 default[20190807t224659] Post is saved to Index: test11asdxxxxxx1asdasd333111111 2019-08-08 06:04:42 default[20190807t230210] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.OkUI/main.go:39 +0x42d 2019-08-08 06:04:43 default[20190807t230210] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.OkUI/main.go:39 +0x42d 2019-08-08 06:05:41 default[20190807t224659] started-service 2019-08-08 06:12:13 default[20190807t231000] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.sxOS/main.go:39 +0x42d 2019-08-08 06:12:15 default[20190807t231000] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.sxOS/main.go:39 +0x42d 2019-08-08 07:17:55 default[20190808t001517] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.DEKl/main.go:41 +0x42d 2019-08-08 07:18:00 default[20190808t001517] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.DEKl/main.go:41 +0x42d 2019-08-08 15:45:29 default[20190808t084311] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.Hr54/main.go:41 +0x42d 2019-08-08 16:09:22 default[20190808t090647] panic: Head http://35.224.157.137:9200/around: dial tcp 35.224.157.137:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.ZkOe/main.go:48 +0x3fe 2019-08-08 16:09:26 default[20190808t090647] panic: Head http://35.224.157.137:9200/around: dial tcp 35.224.157.137:9200: i/o timeout goroutine 1 [running]: main.main() /workspace/staging.ZkOe/main.go:48 +0x3fe 2019-08-08 16:28:50 default[20190808t092603] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.mMyN/main.go:41 +0x42d 2019-08-08 16:55:56 default[20190808t095322] panic: no Elasticsearch node available goroutine 1 [running]: main.main() /workspace/staging.tnrs/main.go:42 +0x42d

MyCode package main import ( "encoding/json" "fmt" "log" "net/http" "reflect" "strconv"

"github.com/pborman/uuid"
elastic "gopkg.in/olivere/elastic.v3"

) const ( INDEX = "around" TYPE = "post" DISTANCE = "200km" // PROJECT_ID = "around-" // BT_INSTANCE = "around-post" // ES_URL = "http://34.66.68.187:9200" )

type Location struct { Lat float64 json:"lat" Lon float64 json:"lon" }

type Post struct { User string json:"user" Message string json:"message" Location Location json:"location" }

func main() { // Create a client client, err := elastic.NewClient(elastic.SetURL(ES_URL), elastic.SetSniff(false)) if err != nil { panic(err) return } //Use the indexExist to check if a specified index exist exists, err := client.IndexExists(INDEX).Do() if err != nil { panic(err) } if !exists { // Create a new index mapping := { "mappings":{ "post":{ "properties":{ "location":{ "type":"geo_point" } } } } } _, err := client.CreateIndex(INDEX).Body(mapping).Do() if err != nil { panic(err) } } fmt.Println("started-service") http.HandleFunc("/post", handlerPost) http.HandleFunc("/search", handlerSearch) log.Fatal(http.ListenAndServe(":8080", nil)) }

func handlerSearch(w http.ResponseWriter, r *http.Request) { fmt.Print("Received one request for search") lat, := strconv.ParseFloat(r.URL.Query().Get("lat"), 64) lon, := strconv.ParseFloat(r.URL.Query().Get("lon"), 64) ran := DISTANCE if val := r.URL.Query().Get("range"); val != "" { ran = val + "km" } fmt.Printf("Search received: %f %f %s\n", lat, lon, ran) //Create a client client, err := elastic.NewClient(elastic.SetURL(ES_URL), elastic.SetSniff(false)) if err != nil { panic(err) return } //Define geo distance query as specified q := elastic.NewGeoDistanceQuery("location") q = q.Distance(ran).Lat(lat).Lon(lon)

searchResult, err := client.Search().
    Index(INDEX).
    Query(q).
    Pretty(true).
    Do()
if err != nil {
    panic(err)
}
fmt.Printf("Query took %d milliseconds\n", searchResult.TookInMillis)
fmt.Printf("Found a total of %d post\n", searchResult.TotalHits())

var typ Post
var ps []Post
for _, item := range searchResult.Each(reflect.TypeOf(typ)) {
    p := item.(Post)
    fmt.Printf("Post by %s: %s at lat %v and lon %v\n", p.User, p.Message, p.Location.Lat, p.Location.Lon)
    ps = append(ps, p)
    if !containsFilteredWords(&p.Message) {
        ps = append(ps, p)
    }
}

js, err := json.Marshal(ps)
if err != nil {
    panic(err)
    return
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Write(js)

}

func handlerPost(w http.ResponseWriter, r *http.Request) { fmt.Println("Received one post request") decoder := json.NewDecoder(r.Body) var p Post if err := decoder.Decode(&p); err != nil { panic(err) return } id := uuid.New() saveToES(&p, id) }

// Save a post to ElasticSearch func saveToES(p *Post, id string) { es_client, err := elastic.NewClient(elastic.SetURL(ESURL), elastic.SetSniff(false)) if err != nil { panic(err) return } // Save it to index , err = es_client.Index(). Index(INDEX). Type(TYPE). Id(id). BodyJson(p). Refresh(true). Do() if err != nil { panic(err) return } fmt.Printf("Post is saved to Index: %s\n", p.Message) }

Question

  1. Where did I do wrong?
  2. Why I receive a lot of info but actually I only post one new data

Thank you so much for helping me!

olivere commented 5 years ago

May I ask which Elasticsearch version you are working against?

JeffreyLiuUSC commented 5 years ago

May I ask which Elasticsearch version you are working against?

I am using elasticsearch-2.3.1.deb on google cloud

olivere commented 5 years ago

Can you check this Wiki page first? https://github.com/olivere/elastic/wiki/Connection-Problems#how-to-figure-out-connection-problems

What’s the output of the Nodes API?

Can you also disable health checks, just to be sure?

JeffreyLiuUSC commented 5 years ago

Thanks for your reply!

I am sorry that I forgot to mention that i think elastic works well on cloud. Reason: 1. type 35.239.234.232:9200 on browser return correct format of data. 2. "curl -s -XGET 'http://35.239.234.232:9200/_nodes/http?pretty=1'" return as expected

However Now I have SetHealthecheck(false) and SetSniff(false) successfully deployed but still same POST/GET error

image
olivere commented 5 years ago

It seems that only the HEAD request is failing? That should point to the IndexExists section of the code. What also irritates me is that it's also using port 9300, which is the transport endpoint, not the REST endpoint.

Not really sure what's happening here TBH.

JeffreyLiuUSC commented 5 years ago

It seems that only the HEAD request is failing? That should point to the IndexExists section of the code. What also irritates me is that it's also using port 9300, which is the transport endpoint, not the REST endpoint.

Not really sure what's happening here TBH.

Maybe I messed up some configuration file when I was trying to solve this issue.... Thanks for helping me. I will do it all again to see if it make some difference. Just one more quick question, do you think google cloud VM instance and Elasticsearch run on different location(one on us-central and one on us-west) has some impact on the errors?

olivere commented 5 years ago

It shouldn't be a big problem. Maybe you can increase timeout values by configuring the client appropriately. ES 2.x and elastic.v3 are rather old, so there e.g. was no context package. Whenever you can, use an up-to-date version of Elasticsearch (7.x at the time of writing this). It's much more stable and bulletproof than 2.x.

JeffreyLiuUSC commented 5 years ago

OK, I will try to re-try it with a newer version of elastic later. Thank you so much for helping me!!! Have a good one!