search

ruippeixotog / ebay-snipe-server

An eBay auction sniping service managed by a RESTful API
MIT License
134 stars 21 forks source link

System detecting wrong end date time auctions #7

Closed iwalucas closed 5 years ago

iwalucas commented 6 years ago

I have just deployed the ebay-snipe-server, and all the auctions I am watching are returning something like that:

{"ended":true,"endingAt":"1970-01-01T00:00:00.000Z",...}

So I cannot place a snipe b/c it says auction has ended.

ruippeixotog commented 6 years ago

@iwalucas I have just tried the server and I had no problem with the /auction/<id> endpoint. Can you tell me one of the IDs that is failing for you and any custom setting you may have in your application.conf?

iwalucas commented 6 years ago

@ruippeixotog here is one I just picked up: 292496978835

this is the output: {"ended":true,"endingAt":"1970-01-01T00:00:00.000Z","location":"Riga, default, Letônia","bidCount":0,"shippingCost":"GBP 5.0","currentBid":null,"id":"292496978835","buyNowPrice":"GBP 7.0","defaultCurrency":"GBP","title":"SI TECH LP Scuba Hose BCD Inflator Dry Suit Standart Fitting Dive 80cm/31.5inch | eBay","seller":{"id":"to_deal","feedback":1409,"positivePercentage":100.0},"thumbnailUrl":"https://i.ebayimg.com/images/i/292496978835-0-1/s-l1000.jpg"}

there are no custom settings

ruippeixotog commented 6 years ago

That's strange... I have just tried getting that auction and I got the right date:

{
    "ended": false,
    "endingAt": "2018-04-23T17:21:19.000+01:00",
    "location": "Riga, default, Latvia",
    "bidCount": 0,
    "shippingCost": "GBP 5.99",
    "currentBid": null,
    "id": "292496978835",
    "buyNowPrice": "GBP 7.49",
    "defaultCurrency": "GBP",
    "title": "SI TECH LP Scuba Hose BCD Inflator Dry Suit Standart Fitting Dive 80cm/31.5inch  | eBay",
    "seller": {
        "id": "to_deal",
        "feedback": 1411,
        "positivePercentage": 100
    },
    "thumbnailUrl": "https://i.ebayimg.com/images/i/292496978835-0-1/s-l1000.jpg"
}

It seems that the date is the only field with wrong information. Are you running this on Windows? What's the locale of your computer?

iwalucas commented 6 years ago

I am running on a Mac, with GMT-3 timezone

ruippeixotog commented 6 years ago

Ok, I'm running on Mac too. The fact that you mentioned "Letônia" while I had "Latvia" at the item location points to a difference in localization on the eBay webpage, which can cause this issue if the date is in another place or in another format. It also makes this hard for me to reproduce it... Can you tell me the output of locale in your shell?

iwalucas commented 6 years ago

sure, here it is:

LANG="en_US.UTF-8"

LC_COLLATE="en_US.UTF-8"

LC_CTYPE="en_US.UTF-8"

LC_MESSAGES="en_US.UTF-8"

LC_MONETARY="en_US.UTF-8"

LC_NUMERIC="en_US.UTF-8"

LC_TIME="en_US.UTF-8"

LC_ALL=

lcd1232 commented 5 years ago

I have the same error:

root@server:~# http localhost:3647/auction/202601112701
HTTP/1.1 200 OK
Content-Length: 439
Content-Type: application/json
Date: Mon, 18 Feb 2019 17:38:10 GMT
Server: akka-http/10.0.9

{
    "bidCount": 0,
    "buyNowPrice": null,
    "currentBid": "EUR 600.0",
    "defaultCurrency": "EUR",
    "ended": true,
    "endingAt": "1970-01-01T00:00:00.000Z",
    "id": "202601112701",
    "location": "Cassino, Италия",
    "seller": {
        "feedback": 44,
        "id": "ice_plug",
        "positivePercentage": 100.0
    },
    "shippingCost": "EUR 39.0",
    "thumbnailUrl": "https://i.ebayimg.com/images/i/202601112701-0-1/s-l1000.jpg",
    "title": "APPLE MACBOOK PRO 13\" RETINA DISPLAY 2015 RAM 8GB SSD 128GB   | eBay"
}
Terr commented 5 years ago

Hi,

I am encountering the same issue when requesting auctions from my local eBay (ebay.nl). The date is correctly returned when getting the same auction via ebay.com.

Like in the US, the ending date is mentioned also mentioned in the .vi-tm-left element, under #LeftSummaryPanel, on the Dutch version. But query("ending-at") keeps returning a None when logging the output in EbayClient.scala.

I don't think I'm at the point where date formatting matters, but I have changed it in my ebay-nl.conf to match the Dutch date formats:

auction-info {
  attributes {
    ending-at = { query: ".vi-tm-left, .lable:contains(\"Afgelopen:\") + .u-flL", date-formats: ["(dd MMM yyyy HH:mm:ss zzz)", "dd-MMM HH:mm"] }
  }
}

The locale of the environment the server is running in is set to nl_NL.UTF-8.

The other data is found and returned just fine. Any idea what could cause only .vi-tm-left to not be found?

Terr commented 5 years ago

I still have no idea what causes the issue, but I have learned two things:

ruippeixotog commented 5 years ago

Hi @Terr! It sounds plausible that ebay.nl CSS queries have flaws, I didn't test the project with any website other than ebay.com.

Do you mind sending me your current ebay-nl.conf and one or more auctions that are currently failing? Once this is fixed I can include the .nl config in the repo, if you want.

Terr commented 5 years ago

Hi, thanks for the reply!

I have pasted my ebay-nl.conf and common.conf in this Gist.

I had to modify common.conf because I failed to override the auction-info.attributes via ebay-nl.conf but maybe I did it wrong. Should it be possible to override those attributes in a site-specific config file?

The new date format pattern is correct I think, but because that's something that I needed to change I might have made a mistake there.

Here are two random auctions that I tried just now and have a few days remaining:

I did not translate all the text patterns in ebay-nl.conf yet, but the ones that I did do are enough to log in and retrieve auctions. After I've come across the other texts that need translation I would be happy to share it in a pull request :)

ruippeixotog commented 5 years ago

Hi @Terr! Sorry for taking so long to reply. I finally got the chance to look at your config and the problem seems to be that the date pattern in auction-info.attributes.ending-at.date-formats does not accept dutch month names like "mei".

Unfortunately scala-scraper, the library that extracts content from HTML in the server, does not support localization currently. I just opened https://github.com/ruippeixotog/scala-scraper/issues/65, feel free to contribute if you find the time for it 🙂

ruippeixotog commented 5 years ago

Closing this issue because I'm not officially supporting ebay.nl (even though I'd like to have a config for it here) and I don't have any reproducible example of this failing in ebay.com.

Terr commented 5 years ago

Thanks for taking the time to investigate.

I'm not sure I can contribute code to scala-scraper but I've subscribed to the issue, in case I can help with testing it.

Should I get a complete and working config for ebay.nl together in the future I will be sure to create a merge request :)