async with aiohttp.ClientSession() as session:

[goodthebest]

@eLvErDe Adam I tried your script and end up getting this error, below is debug

any idea?

[mtompkins]

i forget how, but if you add a console output of the algo variable I'll let you know what it has just to confirm

[eLvErDe]

https://github.com/eLvErDe/yiimp/commit/0b72b984e6e207656f54e4765b48465185381238

[mtompkins]

Same

[eLvErDe]

Well sadly I won't be able to do more. I'm not using this algo variable AT ALL in this page, so without having the error_log with proper traceback...

[mtompkins]

Shit it may be permissions - I just realized I created the new file as root.

[eLvErDe]

I don't think so, it's still readable unless you're running a really weird umask.

Hold on, just saw somethings....

[mtompkins]

Well it wasn't permissions - that's confirmed

[mtompkins]

Also when I get a chance I'll set the VM to capture the php errors - that'll help

[mtompkins]

used this

ini_set("log_errors", 1);
ini_set("error_log", "/tmp/php-error.log");

and didn't see any errors when refreshing

[eLvErDe]

https://github.com/eLvErDe/yiimp/commit/4c5e517e2132907040666aa0c5b897e33673bdee

[mtompkins]

Progress!! This is accurate

[eLvErDe]

So do you think it's working ?

[mtompkins]

Checking passing arguments now ... ?algo=all works checking other

[eLvErDe]

Wait, I'd like to do a small modification first

[mtompkins]

Yep - you nailed it!

[mtompkins]

I'll update the config and revert on the scraper -> discord

[mtompkins]

ok - I presume ?algo=all URL isn't going to change

[eLvErDe]

https://github.com/eLvErDe/yiimp/commit/543ac7541436920f0b60d1ce6ce1ca4f02aed6b7

[mtompkins]

I think it's ok but there is some weirdness to it - which may be memcache. If I go to ?algo=all - lists correctly, then go to a ?algo=neoscrypt it doesn't always update correctly. Not sure but I can live with it. I don't think anyone is going to switch that way.

[eLvErDe]

This is a final version I'd like to sent to tpruvot if you can do the following tests. Please also check the proper algorithm are shown in the page headers (inside parenthesis)

Nothing Nothing but change the algo by clicking on a different algorithm from the front page ?algo= (what happens ?) ?algo (what happens ?) ?algo=all ?algo=x17 ?algo=x17,neoscrypt,x11 ?algo=xé@17 (should be stripped down to x17)

Thanks !

[mtompkins]

Doing this in succession . . .

1. Nothing | results in a random algo (like current implementation)
2. Nothing w/click | results in the clicked algo
3. ?algo= | no change on the page - displays #2
4. ?algo | no change on the page - displays #2
5. ?algo=all | displays all
6. ?algo=x17 | displays x17
7. ?algo=x17,neoscrypt,x11 | displays the 3 correctly
8. ?algo=xé@17 | displays x17 as intended

Looks good

[mtompkins]

Bot is updated and active - will confirm findings after block - thanks for your work on this.

[mtompkins]

Hit a block I believe it should have caught - no announcement to discord. Will confirm on next.

[eLvErDe]

PR sent to tpruvot: https://github.com/tpruvot/yiimp/pull/256

[mtompkins]

Ok - a ton of SSL errors

Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: ERROR    [poll_yiimp_events        ] Exception occurred: ClientOSError: [Errno 111] Cannot connect to host some.yiimppool.com:443 ssl:True [Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]]
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 575, in _create_connection
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     server_hostname=hinfo['hostname'] if sslcontext else None)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     raise exceptions[0]
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     yield from self.sock_connect(sock, address)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/selector_events.py", line 402, in sock_connect
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     return (yield from fut)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     yield self  # This tells Task to wait for completion.
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     future.result()
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     raise self._exception
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/selector_events.py", line 432, in _sock_connect_cb
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     raise OSError(err, 'Connect call failed %s' % (address,))
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: ConnectionRefusedError: [Errno 111] Connect call failed ('35.196.236.102', 443)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: The above exception was the direct cause of the following exception:
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 307, in connect
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     transport, proto = yield from self._create_connection(req)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 598, in _create_connection
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     (req.host, req.port, exc.strerror)) from exc
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: aiohttp.errors.ClientOSError: [Errno 111] Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: The above exception was the direct cause of the following exception:
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/local/bin/yiimp-blocks-found-to-discord.py", line 126, in poll_yiimp_events
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     resp = await session.get(url)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 499, in __await__
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     resp = yield from self._coro
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 174, in _request
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     conn = yield from self._connector.connect(req)
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 317, in connect
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]:     .format(key, exc.strerror)) from exc
Apr 12 15:46:58 yiimp-blocks-found-to-discord[19179]: aiohttp.errors.ClientOSError: [Errno 111] Cannot connect to host some.yiimppool.com:443 ssl:True [Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]]
Apr 12 15:47:02 yiimp-blocks-found-to-discord[19179]: INFO     [refresh_st_exc_markets   ] Refreshed
Apr 12 15:47:02 yiimp-blocks-found-to-discord[19179]: INFO     [refresh_cryptopia_markets] Refreshed
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: ERROR    [poll_yiimp_events        ] Exception occurred: ClientOSError: [Errno 111] Cannot connect to host some.yiimppool.com:443 ssl:True [Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]]
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 575, in _create_connection
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     server_hostname=hinfo['hostname'] if sslcontext else None)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 695, in create_connection
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     raise exceptions[0]
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 682, in create_connection
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     yield from self.sock_connect(sock, address)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/selector_events.py", line 402, in sock_connect
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     return (yield from fut)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     yield self  # This tells Task to wait for completion.
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     future.result()
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     raise self._exception
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3.5/asyncio/selector_events.py", line 432, in _sock_connect_cb
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     raise OSError(err, 'Connect call failed %s' % (address,))
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: ConnectionRefusedError: [Errno 111] Connect call failed ('35.196.236.102', 443)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: The above exception was the direct cause of the following exception:
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 307, in connect
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     transport, proto = yield from self._create_connection(req)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 598, in _create_connection
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     (req.host, req.port, exc.strerror)) from exc
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: aiohttp.errors.ClientOSError: [Errno 111] Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: The above exception was the direct cause of the following exception:
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: Traceback (most recent call last):
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/local/bin/yiimp-blocks-found-to-discord.py", line 126, in poll_yiimp_events
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     resp = await session.get(url)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 499, in __await__
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     resp = yield from self._coro
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/client.py", line 174, in _request
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     conn = yield from self._connector.connect(req)
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:   File "/usr/lib/python3/dist-packages/aiohttp/connector.py", line 317, in connect
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]:     .format(key, exc.strerror)) from exc
Apr 12 15:47:03 yiimp-blocks-found-to-discord[19179]: aiohttp.errors.ClientOSError: [Errno 111] Cannot connect to host some.yiimppool.com:443 ssl:True [Can not connect to some.yiimppool.com:443 [Connect call failed ('35.196.236.102', 443)]]

[mtompkins]

35.196.236.102 isn't me (some.yiimppool.com) .. presume it's discord?

[eLvErDe]

No it's coming from the "poll_yiimp_events" coroutine. Are you sure to have https:// in the URL ? That's weird. I have no idea what could be 35.196.236.102

[mtompkins]

Yes it is correct. I also tried a curl to the ?algo=all and it comes back fine. A curl to the webhook addy also returns correctly.

[eLvErDe]

ConnectionRefusedError: [Errno 111] Connect call failed ('35.196.236.102', 443)

Well that's definetely the address you are trying to connect on....

[eLvErDe]

You did not have this issue before right ? Can you remove ?algo=all from the URL and confirm it's working again ?

[eLvErDe]

That might be again related to the prehistoric version of aiohttp you are using (not being able to handle query param as part of the URL itself)

[mtompkins]

Correct, and it never worked because it always had the wrong algo. I agree and it was my intent to try the newer aiohttp and see if that solved it.

[eLvErDe]

So removing algo=all make the aiohttp call works ?

[mtompkins]

I don't know and won't be able to tell. It brings us back to where we started with no blocks on an algo with no action.

[mtompkins]

Testing now with current v. of aiohttp - will revert on block.

[eLvErDe]

Well no we are not ! We patched YIIMP code to avoid a proper page to be scrapped.

Problem is that at the moment I don't understand your testing.

• Do you have an aiohttp traceback in poll_yiimp_events when removing ?algo=all from the URL ?
• Can you double check removing ?algo=all from the URL make the traceback DISAPPEAR ?

[mtompkins]

Removing the parameter puts us right back to where we started, just as in the first test you asked me to run. We arrive at a "random" page as I explicitly indicated above.

1. Nothing | results in a random algo (like current implementation)

The page does not default to all without a parameter, so again, we are right back to the beginning of this experiment. I'm on the new aiohttp and will revert with that outcome.

[eLvErDe]

Yes I get that, but if the aiohttp traceback is gone when you remove ?algo=all, it means your aiohttp cannot handle that and that would be easy to fix.

[mtompkins]

How long should it be from a posted new block until the python script picks it up? Is there a poll period? I had a block at :03 and there was no evidence of it (no error either) in watching the logs for 7 minutes. This is with the master branch and most current aiohttp

Once I understand the poll period I'll have a look at this recent work again.

[eLvErDe]

It's nearly live but the block has to appear while the script is already running. If you started the script and the block was found already it won't find it

[mtompkins]

OK - that's what I anticipated. The script was started before the block was found, by minutes. I've reverted to the old aiohttp and your current changes. I'll wait for a block, capture out, then try again without the parameter.

[eLvErDe]

The easiest way would be to give me your pool address ;-) By private email if you prefer.

[mtompkins]

I appreciate that, unfortunately isn't an option atm.

[eLvErDe]

Which version of aiohttp are you running ? Default one from Ubuntu xenial? I'll start my script in a docker container to confirm if it can work with such an old version

[mtompkins]

Now on:
python3-bs4 is already the newest version (4.4.1-1). python3-aiohttp is already the newest version (0.20.2-1)

[eLvErDe]

For the record I created the following docker file:

FROM ubuntu:xenial

MAINTAINER Adam Cecile <acecile@le-vert.net>

ENV TERM xterm
ENV DEBIAN_FRONTEND=noninteractive

# Install dependencies
RUN apt update && apt -y -o 'Dpkg::Options::=--force-confdef' -o 'Dpkg::Options::=--force-confold' --no-install-recommends install wget ca-certificates python3-bs4 python3-html5lib python3-aiohttp && rm -rf /var/lib/apt/lists/*

# Install script from legacy branch
RUN wget https://raw.githubusercontent.com/eLvErDe/yiimp-blocks-found-to-discord/old-aiohttp-compat/yiimp-blocks-found-to-discord.py -O /root/yiimp-blocks-found-to-discord.py && chmod 0755 /root/yiimp-blocks-found-to-discord.py

Built using:

docker build -t test-yiimp-xenial . -f Dockerfile.ubuntu.xenial --no-cache

Run using:

docker run -it --rm test-yiimp-xenial /root/yiimp-blocks-found-to-discord.py -p https://pool.ionik.fr/site/found_results?algo=all -d https://discordapp.com/api/webhooks/123/345

And it's polling and finding blocks just fine however, posting to Discords is still fucked up, thanks to aiohttp 0.20.

[mtompkins]

Yea I figured (purely guessing) it has something to do with TLS and this older aiohttp. Did you test your master branch and current aiohttp because Discord didn't work on that for me (unless I missed something)?

[eLvErDe]

It's running on my Debian Buster with aiohttp 3.0.1-1 just fine. I'm pretty sure I was running 2.x when writing this script

[mtompkins]

OK I'll try 3.0.1

[eLvErDe]

Not needed I'm fixing remaining issues with 0.20 in the legacy branch.