search

dwyl / esta

:mag: Simple + Fast ElasticSearch Node.js client. Beginner-friendly defaults & Heroku support :white_check_mark: :rocket:
48 stars 10 forks source link

Is there a method to return ALL the ids in the index? #93

Open nelsonic opened 8 years ago

nelsonic commented 8 years ago

http://stackoverflow.com/questions/17497075/efficient-way-to-retrieve-all-ids-in-elasticsearch

nelsonic commented 8 years ago 
curl http://localhost:9200/twitter/tweet/_search?pretty=true -d '
{ 
    "query" : { 
        "match_all" : {} 
    },
    "fields": []
}
'

Gives:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 56,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "894117058953",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559463492466782200",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559463323897307140",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559464056734896100",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559464187764949000",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559464082508496900",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559463500398211100",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559464234510471200",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559464087571410940",
      "_score" : 1.0
    }, {
      "_index" : "twitter",
      "_type" : "tweet",
      "_id" : "559463492634562560",
      "_score" : 1.0
    } ]
  }
}
nelsonic commented 8 years ago

If we then combine this with a scroll: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html

curl http://localhost:9200/twitter/tweet/_search?scroll=5m&pretty=true -d '
{ 
    "query" : { 
        "match_all" : {} 
    },
    "fields": []
}
'

Returns:

{
  "_scroll_id": "cXVlcnlUaGVuRmV0Y2g7NTszNzE6UUgxVWhEYy1UZUNLTU10RWpPczdvUTszNzU6UUgxVWhEYy1UZUNLTU10RWpPczdvUTszNzI6UUgxVWhEYy1UZUNLTU10RWpPczdvUTszNzM6UUgxVWhEYy1UZUNLTU10RWpPczdvUTszNzQ6UUgxVWhEYy1UZUNLTU10RWpPczdvUTswOw==",
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 56,
    "max_score": 1,
    "hits": [
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "894117058953",
        "_score": 1,
        "_source": {
          "message": "My Random Message"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559463492466782200",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:31:05 +0000 2015",
          "id_str": "559463492466782208",
          "text": "Photo: \"Shuu acknowledges the amazing magnificence of Ken Kaneki. (Rate: 10 out of 10)\"\u00a0 SO. ACCURATE.\u00a0 http:\/\/t.co\/u0RF8t1oQy",
          "source": "<a href=\"http:\/\/www.tumblr.com\/\" rel=\"nofollow\">Tumblr<\/a>",
          "truncated": false,
          "user": {
            "id": 156514410,
            "id_str": "156514410",
            "name": "Kat.Sq",
            "screen_name": "KatDuterre",
            "location": "Canada",
            "url": "http:\/\/vkontakte.ru\/catrice",
            "description": null,
            "protected": false,
            "verified": false,
            "followers_count": 19,
            "friends_count": 88,
            "listed_count": 1,
            "favourites_count": 36,
            "statuses_count": 875,
            "created_at": "Thu Jun 17 04:07:44 +0000 2010",
            "utc_offset": -21600,
            "time_zone": "Central Time (US & Canada)",
            "geo_enabled": true,
            "lang": "ru",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "F5F4EB",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/707951865\/a7569f944abdcc19fdff4520170b33bc.jpeg",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/707951865\/a7569f944abdcc19fdff4520170b33bc.jpeg",
            "profile_background_tile": true,
            "profile_link_color": "12B31C",
            "profile_sidebar_border_color": "FFFFFF",
            "profile_sidebar_fill_color": "F6FFD1",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/3450194233\/fa8dcccce07a4f87b465af8366d72fe9_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/3450194233\/fa8dcccce07a4f87b465af8366d72fe9_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/156514410\/1364617701",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [
              {
                "url": "http:\/\/t.co\/u0RF8t1oQy",
                "expanded_url": "http:\/\/tmblr.co\/ZNHhPx1bf8JiG",
                "display_url": "tmblr.co\/ZNHhPx1bf8JiG",
                "indices": [
                  104,
                  126
                ]
              }
            ],
            "user_mentions": [

            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221465590"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559463323897307140",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:30:25 +0000 2015",
          "id_str": "559463323897307136",
          "text": "BuyNeu: Get or create amazing costumes of your favorite characters! http:\/\/t.co\/VAWQ7m8DNn",
          "source": "<a href=\"http:\/\/dlvr.it\" rel=\"nofollow\">dlvr.it<\/a>",
          "truncated": false,
          "user": {
            "id": 2202251011,
            "id_str": "2202251011",
            "name": "Lorretta Schrupp",
            "screen_name": "LorrettaSchrupp",
            "location": "",
            "url": "http:\/\/comic.stylinghub.com\/",
            "description": "I love comic, I even created a site just to share all about comics. #comics",
            "protected": false,
            "verified": false,
            "followers_count": 557,
            "friends_count": 11,
            "listed_count": 18,
            "favourites_count": 0,
            "statuses_count": 52095,
            "created_at": "Tue Nov 19 01:03:09 +0000 2013",
            "utc_offset": null,
            "time_zone": null,
            "geo_enabled": false,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "0000FF",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/378800000123514465\/ace7ed8a6d9541fb171ecc4ece263a2d.jpeg",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/378800000123514465\/ace7ed8a6d9541fb171ecc4ece263a2d.jpeg",
            "profile_background_tile": true,
            "profile_link_color": "0000FF",
            "profile_sidebar_border_color": "C0DEED",
            "profile_sidebar_fill_color": "DDEEF6",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/378800000788897977\/96a737275686656256d76f828694ff11_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/378800000788897977\/96a737275686656256d76f828694ff11_normal.jpeg",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [
              {
                "url": "http:\/\/t.co\/VAWQ7m8DNn",
                "expanded_url": "http:\/\/dlvr.it\/8DzdJ9",
                "display_url": "dlvr.it\/8DzdJ9",
                "indices": [
                  68,
                  90
                ]
              }
            ],
            "user_mentions": [

            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221425400"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559464056734896100",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:33:20 +0000 2015",
          "id_str": "559464056734896128",
          "text": "Just turning off your phone for a while is an amazing thing",
          "source": "<a href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\">Twitter for iPhone<\/a>",
          "truncated": false,
          "user": {
            "id": 945003464,
            "id_str": "945003464",
            "name": "Kellianne",
            "screen_name": "kell_laurieee",
            "location": "",
            "url": null,
            "description": "\u029f\u026a\u0493\u1d07s \u026as \u029f\u026a\u1d0b\u1d07 \u1d00 \u1d04\u1d00\u1d0d\u1d07\u0280\u1d00, \u0493\u1d0f\u1d04\u1d1cs \u1d0f\u0274 \u1d21\u029c\u1d00\u1d1b \u026as \u026a\u1d0d\u1d18\u1d0f\u1d1b\u1d00\u0274\u1d1b & \u028f\u1d0f\u1d1c'\u029f\u029f \u1d04\u1d00\u1d18\u1d1b\u1d1c\u0280\u1d07 \u026a\u1d1b \u1d18\u1d07\u0280\u0493\u1d07\u1d04\u1d1b\u029f\u028f\u2661 \u0492\u03b1\u01a8\u043d\u03b9\u03c3\u0438\u2661D\u0280\u03b5\u03b1\u028d",
            "protected": false,
            "verified": false,
            "followers_count": 506,
            "friends_count": 615,
            "listed_count": 3,
            "favourites_count": 12817,
            "statuses_count": 5175,
            "created_at": "Tue Nov 13 02:08:29 +0000 2012",
            "utc_offset": -18000,
            "time_zone": "Eastern Time (US & Canada)",
            "geo_enabled": false,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "642D8B",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/721240489\/36f937d9f37c3502522611bdfe5b452f.jpeg",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/721240489\/36f937d9f37c3502522611bdfe5b452f.jpeg",
            "profile_background_tile": true,
            "profile_link_color": "FF0000",
            "profile_sidebar_border_color": "FFFFFF",
            "profile_sidebar_fill_color": "7AC3EE",
            "profile_text_color": "3D1957",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/557020799564263428\/YVHg87d-_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/557020799564263428\/YVHg87d-_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/945003464\/1420600283",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [

            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221600122"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559464187764949000",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:33:51 +0000 2015",
          "id_str": "559464187764948993",
          "text": "\u2665 @camerondallas, I hope you're having a good day baby! if u're reading this I want you to smile\u30c4 bc you are amazing &amp; I love you \u2665 x519",
          "source": "<a href=\"http:\/\/twitter.com\" rel=\"nofollow\">Twitter Web Client<\/a>",
          "truncated": false,
          "user": {
            "id": 865191630,
            "id_str": "865191630",
            "name": "\u2727\u2218* Cameron\u2727\u2218",
            "screen_name": "oh5SOSorgasms",
            "location": "Neverland",
            "url": null,
            "description": "\u275dand i'm thinking about how people fall in love in mysterious ways\u275e1\/midnight red + colton follows",
            "protected": false,
            "verified": false,
            "followers_count": 833,
            "friends_count": 1425,
            "listed_count": 2,
            "favourites_count": 252,
            "statuses_count": 2614,
            "created_at": "Sat Oct 06 19:37:49 +0000 2012",
            "utc_offset": 3600,
            "time_zone": "Madrid",
            "geo_enabled": false,
            "lang": "es",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "C0DEED",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/556601770495848448\/bSWqQCBf.png",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/556601770495848448\/bSWqQCBf.png",
            "profile_background_tile": true,
            "profile_link_color": "0084B4",
            "profile_sidebar_border_color": "000000",
            "profile_sidebar_fill_color": "DDEEF6",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/557980762171924481\/PeOlDETX_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/557980762171924481\/PeOlDETX_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/865191630\/1421868541",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [
              {
                "screen_name": "camerondallas",
                "name": "Cameron Dallas",
                "id": 405728790,
                "id_str": "405728790",
                "indices": [
                  2,
                  16
                ]
              }
            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221631362"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559464082508496900",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:33:26 +0000 2015",
          "id_str": "559464082508496896",
          "text": "Just thought of something potentially amazing. @berner415 . how about some marketing with Brown's WR josh Gordon?",
          "source": "<a href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\">Twitter for Android<\/a>",
          "truncated": false,
          "user": {
            "id": 2490411656,
            "id_str": "2490411656",
            "name": "Harold ",
            "screen_name": "TrvpHvrold",
            "location": "San Diego",
            "url": null,
            "description": "Just an 18 year old chillin playing league and listening to music. #tgod #uzi #hbk",
            "protected": false,
            "verified": false,
            "followers_count": 79,
            "friends_count": 223,
            "listed_count": 0,
            "favourites_count": 4958,
            "statuses_count": 1353,
            "created_at": "Sun May 11 21:15:52 +0000 2014",
            "utc_offset": -25200,
            "time_zone": "Arizona",
            "geo_enabled": false,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "1A1B1F",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/552664587917459456\/j_U-qq02.png",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/552664587917459456\/j_U-qq02.png",
            "profile_background_tile": true,
            "profile_link_color": "2FC2EF",
            "profile_sidebar_border_color": "FFFFFF",
            "profile_sidebar_fill_color": "252429",
            "profile_text_color": "666666",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/470662702394642432\/Pd37cmW7_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/470662702394642432\/Pd37cmW7_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/2490411656\/1421029192",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [
              {
                "screen_name": "berner415",
                "name": "#20LIGHTS",
                "id": 22312032,
                "id_str": "22312032",
                "indices": [
                  47,
                  57
                ]
              }
            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221606267"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559463500398211100",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:31:07 +0000 2015",
          "id_str": "559463500398211072",
          "text": "Hi @Harry_Styles \nYou're amazing\ntalented\nthoughtful\ncaring\nkind\nthe list is endless\u30c4\nHope that made you smile\nfollow me? \nI love u x30,828",
          "source": "<a href=\"http:\/\/twitter.com\/#!\/download\/ipad\" rel=\"nofollow\">Twitter for iPad<\/a>",
          "truncated": false,
          "user": {
            "id": 1872033427,
            "id_str": "1872033427",
            "name": "lotte loves harry",
            "screen_name": "harrehstrokes",
            "location": "cheshire ",
            "url": "https:\/\/twitter.com\/harrehstrokes\/status\/549266660537151488",
            "description": "it costs nothing to be nice",
            "protected": false,
            "verified": false,
            "followers_count": 1301,
            "friends_count": 962,
            "listed_count": 45,
            "favourites_count": 5855,
            "statuses_count": 60397,
            "created_at": "Mon Sep 16 15:13:40 +0000 2013",
            "utc_offset": 3600,
            "time_zone": "Amsterdam",
            "geo_enabled": true,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "C0DEED",
            "profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
            "profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
            "profile_background_tile": false,
            "profile_link_color": "0084B4",
            "profile_sidebar_border_color": "C0DEED",
            "profile_sidebar_fill_color": "DDEEF6",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/554322183779282945\/3ZfS1Oai_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/554322183779282945\/3ZfS1Oai_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/1872033427\/1420995683",
            "default_profile": true,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [
              {
                "screen_name": "Harry_Styles",
                "name": "Harry Styles.",
                "id": 181561712,
                "id_str": "181561712",
                "indices": [
                  3,
                  16
                ]
              }
            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221467481"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559464234510471200",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:34:02 +0000 2015",
          "id_str": "559464234510471168",
          "text": "@rcollete I see the dresses amazing collection! Nices shots! Versace knows how makes women godness. Thanks for that.",
          "source": "<a href=\"http:\/\/twitter.com\/download\/android\" rel=\"nofollow\">Twitter for Android<\/a>",
          "truncated": false,
          "in_reply_to_status_id": 5.5946160032816e+17,
          "in_reply_to_status_id_str": "559461600328155136",
          "in_reply_to_user_id": 86361893,
          "in_reply_to_user_id_str": "86361893",
          "in_reply_to_screen_name": "rcollete",
          "user": {
            "id": 59805224,
            "id_str": "59805224",
            "name": "Jana\u00edra Fran\u00e7a",
            "screen_name": "JanairaFranca",
            "location": "Brazil",
            "url": null,
            "description": "Mulher imperfeita, real e curiosa #Marketing #DigitalMarketing #Communication #SocialMedia \u2605\u2605\u2605\u2605\u2605 S\u00e3o Paulo",
            "protected": false,
            "verified": false,
            "followers_count": 570,
            "friends_count": 401,
            "listed_count": 11,
            "favourites_count": 757,
            "statuses_count": 5783,
            "created_at": "Fri Jul 24 15:10:43 +0000 2009",
            "utc_offset": -7200,
            "time_zone": "Brasilia",
            "geo_enabled": true,
            "lang": "pt",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "F2F7F7",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/534769366550605824\/nM8g64gQ.jpeg",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/534769366550605824\/nM8g64gQ.jpeg",
            "profile_background_tile": false,
            "profile_link_color": "0725BA",
            "profile_sidebar_border_color": "FFFFFF",
            "profile_sidebar_fill_color": "DDFFCC",
            "profile_text_color": "333333",
            "profile_use_background_image": false,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/559458053834428418\/KTtYakOD_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/559458053834428418\/KTtYakOD_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/59805224\/1416334434",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [
              {
                "screen_name": "rcollete",
                "name": "Collete, Ralf A.L.L ",
                "id": 86361893,
                "id_str": "86361893",
                "indices": [
                  0,
                  9
                ]
              }
            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221642507"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559464087571410940",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:33:27 +0000 2015",
          "id_str": "559464087571410945",
          "text": "@jg_28 thanks Jackie for donating, the support has been amazing\ud83d\ude0a",
          "source": "<a href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\">Twitter for iPhone<\/a>",
          "truncated": false,
          "in_reply_to_status_id": 5.5945990366089e+17,
          "in_reply_to_status_id_str": "559459903660892160",
          "in_reply_to_user_id": 297074652,
          "in_reply_to_user_id_str": "297074652",
          "in_reply_to_screen_name": "jg_28",
          "user": {
            "id": 701721223,
            "id_str": "701721223",
            "name": "Vic Kane",
            "screen_name": "VicGarbutt",
            "location": "Newtownabbey, Northern Ireland",
            "url": null,
            "description": "Life's for livin, laughin & lovin. Mummy to gorgeous sons Archie & Teddie, love my job in a boys school, live for fashion & food!",
            "protected": false,
            "verified": false,
            "followers_count": 346,
            "friends_count": 986,
            "listed_count": 4,
            "favourites_count": 179,
            "statuses_count": 292,
            "created_at": "Tue Jul 17 20:27:47 +0000 2012",
            "utc_offset": null,
            "time_zone": null,
            "geo_enabled": true,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "C0DEED",
            "profile_background_image_url": "http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
            "profile_background_image_url_https": "https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png",
            "profile_background_tile": false,
            "profile_link_color": "0084B4",
            "profile_sidebar_border_color": "C0DEED",
            "profile_sidebar_fill_color": "DDEEF6",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/489867527762489345\/DmWV7B94_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/489867527762489345\/DmWV7B94_normal.jpeg",
            "profile_banner_url": "https:\/\/pbs.twimg.com\/profile_banners\/701721223\/1405685799",
            "default_profile": true,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [

            ],
            "user_mentions": [
              {
                "screen_name": "jg_28",
                "name": "Jack Gourley",
                "id": 297074652,
                "id_str": "297074652",
                "indices": [
                  0,
                  6
                ]
              }
            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221607474"
        }
      },
      {
        "_index": "twitter",
        "_type": "tweet",
        "_id": "559463492634562560",
        "_score": 1,
        "_source": {
          "created_at": "Sun Jan 25 21:31:05 +0000 2015",
          "id_str": "559463492634562561",
          "text": "Great job by Pastor Buddy and Alicia sharing about the amazing things God is doing at the Abba Center! http:\/\/t.co\/VU8SdINxvc",
          "source": "<a href=\"http:\/\/www.facebook.com\/twitter\" rel=\"nofollow\">Facebook<\/a>",
          "truncated": false,
          "user": {
            "id": 965649199,
            "id_str": "965649199",
            "name": "Core Church",
            "screen_name": "CoreChurchAC",
            "location": "Broken Arrow, OK",
            "url": "http:\/\/corechurch.com",
            "description": "At the core of who we are is loving God and loving people.",
            "protected": false,
            "verified": false,
            "followers_count": 37,
            "friends_count": 0,
            "listed_count": 0,
            "favourites_count": 0,
            "statuses_count": 593,
            "created_at": "Fri Nov 23 07:09:56 +0000 2012",
            "utc_offset": -18000,
            "time_zone": "Eastern Time (US & Canada)",
            "geo_enabled": false,
            "lang": "en",
            "contributors_enabled": false,
            "is_translator": false,
            "profile_background_color": "FAFDFF",
            "profile_background_image_url": "http:\/\/pbs.twimg.com\/profile_background_images\/378800000084262591\/194993eca6ac03b108b1a6e03659b703.jpeg",
            "profile_background_image_url_https": "https:\/\/pbs.twimg.com\/profile_background_images\/378800000084262591\/194993eca6ac03b108b1a6e03659b703.jpeg",
            "profile_background_tile": true,
            "profile_link_color": "CF9D06",
            "profile_sidebar_border_color": "000000",
            "profile_sidebar_fill_color": "EFEFEF",
            "profile_text_color": "333333",
            "profile_use_background_image": true,
            "profile_image_url": "http:\/\/pbs.twimg.com\/profile_images\/2884638944\/8985447c0a3d4ad207639076ad141ae3_normal.jpeg",
            "profile_image_url_https": "https:\/\/pbs.twimg.com\/profile_images\/2884638944\/8985447c0a3d4ad207639076ad141ae3_normal.jpeg",
            "default_profile": false,
            "default_profile_image": false,
            "following": null,
            "follow_request_sent": null,
            "notifications": null
          },
          "retweet_count": 0,
          "favorite_count": 0,
          "entities": {
            "hashtags": [

            ],
            "trends": [

            ],
            "urls": [
              {
                "url": "http:\/\/t.co\/VU8SdINxvc",
                "expanded_url": "http:\/\/fb.me\/1Uqr39fwX",
                "display_url": "fb.me\/1Uqr39fwX",
                "indices": [
                  103,
                  125
                ]
              }
            ],
            "user_mentions": [

            ],
            "symbols": [

            ]
          },
          "favorited": false,
          "retweeted": false,
          "possibly_sensitive": false,
          "filter_level": "low",
          "lang": "en",
          "timestamp_ms": "1422221465630"
        }
      }
    ]
  }
}
nelsonic commented 8 years ago

Apparently we're going to need to use from/size to increase the number of hits above 10: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html

raed667 commented 7 years ago

@nelsonic : Does the client SEARCH support size and from parameters ?

nelsonic commented 7 years ago

@RaedsLab not yet... if you have time to add, please go for it! 👍 (P.S: I can make you the maintainer if you're interested...) Only two of our projects are using ElasticSearch right now (the rest are all PostgreSQL) so I'm a bit out of the game with ES.

CC: @SimonLab

raed667 commented 7 years ago

@nelsonic I have already submitted a PR #106 for this feature, which also closes #91

I have just started learning ES (3 days now :smile: ) so I might not be the most qualified person to maintain a client.

nelsonic commented 7 years ago

@RaedsLab thank you so much for the Pull Request!! 🎉 I cannot tell you how excited we are to see people discovering our repos/projects and finding/making the time to improve them! 😍

(perhaps counterintuitively) the fact that you are a "beginner" (at ElasticSearch) makes you the perfect person to read through, extend and maintain this package! You are only "new" to a subject once. Having a fresh perspective means you can continuously ask the question: "did I understand this the first time I saw it?" and if not, then an improvement is required. in which case a new issue is created in the "backlog" for why it was not clear.

See: https://en.wikipedia.org/wiki/Shoshin (Beginner's Mind. what we aspire to...!)