Open pohmelie opened 8 years ago
FTP over SSH
yes, it's just tunneling...
i was thinking of FTPS - actually this might be quite simple, but as usually the main problem is finding free time for doing things (especially with a full-time job)
:+1: Any updates on this? Happy to help out.
@creatorrr, I tried out ssl module:
import asyncio
import ssl
req = b"GET / HTTP/1.1\r\nHost: www.python.org\r\n\r\n"
async def flush(outgoing, writer):
if outgoing.pending:
writer.write(outgoing.read())
await writer.drain()
async def foo():
BLOCK_SIZE = 8192
# ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
ssl_context.check_hostname = False
incoming = ssl.MemoryBIO()
outgoing = ssl.MemoryBIO()
ssl_object = ssl_context.wrap_bio(incoming, outgoing)
reader, writer = await asyncio.open_connection("python.org", 443)
while True:
try:
ssl_object.do_handshake()
break
except ssl.SSLWantReadError:
await flush(outgoing, writer)
incoming.write(await reader.read(BLOCK_SIZE))
ssl_object.write(req)
await flush(outgoing, writer)
data = None
while True:
try:
data = ssl_object.read(BLOCK_SIZE)
print(data)
except ssl.SSLWantReadError:
await flush(outgoing, writer)
await flush(outgoing, writer)
data = await reader.read(BLOCK_SIZE)
# print("->", data)
if not data:
break
incoming.write(data)
writer.close()
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(foo())
But it's pretty annoying… also, I was thinking about who will need this, since I did not find any FTPS server online. Feel free to implement this functionality is you have enough courage and time :wink:
Haha, thanks for looking into this, @pohmelie! I will try and see if I can get things to work and create a patch.
I did not find any FTPS server online
https://github.com/aio-libs/aioftp/pull/81 — works with my private FTPS.
cc @pohmelie
Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.
Unfortunately, there is an asyncio bug, which will come out when your path io is slower than network io. Solution for this (except wait for fix) is own wrapper, like present above.
@pohmelie, for the example code you provided above, I import uvloop and the example code can run without any errors, seems uvloop can fix this issue?
@markshhsu, it looks like yes.
Just to note that I'm failing to connect to an SFTP server.
Trying to form a connection like this:
async with aioftp.Client.context(
host=settings.host,
port=settings.port,
user=settings.username,
password=settings.password.get_secret_value(),
ssl=True,
socket_timeout=10,
path_timeout=10,
) as client:
logger.info("Connected.")
logger.info(str(await client.list()))
which results in this error:
async with aioftp.Client.context(
File "~/miniconda3/envs/fetch/lib/python3.8/contextlib.py", line 171, in __aenter__
return await self.gen.__anext__()
File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 1199, in context
await client.connect(host, port)
File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 604, in connect
await super().connect(host, port)
File "~/miniconda3/envs/fetch/lib/python3.8/site-packages/aioftp/client.py", line 131, in connect
reader, writer = await self._open_connection(host, port)
File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/streams.py", line 52, in open_connection
transport, _ = await loop.create_connection(
File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
transport, protocol = await self._create_connection_transport(
File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
await waiter
File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "~/miniconda3/envs/fetch/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
self._sslobj.do_handshake()
File "~/miniconda3/envs/fetch/lib/python3.8/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)
If I set the ssl
argument to ssl.SSLContext(ssl.PROTOCOL_TLS)
then I see a message
Using selector: EpollSelector
but nothing happens and I have to interrupt.
I can connect to the server with the same information when using pysftp
and disabling the SSH known hosts check.
cnx_opts = pysftp.CnOpts()
cnx_opts.hostkeys = None
with pysftp.Connection(
host=settings.host,
port=settings.port,
username=settings.username,
password=settings.password.get_secret_value(),
cnopts=cnx_opts,
) as sftp:
logger.info("Connected.")
logger.info(sftp.pwd)
logger.info(sftp.listdir())
Do you have any advice on what I'm doing wrong?
:thinking: indeed the distinction between SFTP and FTP over SSH (which you demonstrated in your comment) was not clear to me. Thank you for the heads up.
FWIW, tried with ssl.create_default_context(), server seems to start, client can't connect...
@maulberto3 need more context.
Started working on a raw test implementation for FTPES, but unfortunately didn't get it working. Here is my attempt in case anybody wants to look into it and may have a suggestion or even solution.
@pantierra As a doc said (https://docs.python.org/3/library/asyncio-eventloop.html?highlight=start_tls#asyncio.loop.start_tls)
Return a new transport instance, that the protocol must start using immediately after the await. The transport instance passed to the start_tls method should never be used again.
I think that is the problem
Good point! It looks like we would need for Python v3.11 for this, which includes this PR that makes things much easier: https://github.com/python/cpython/pull/91453.
Any updates on explicit ftps mode?
@antonio-hickey No updates. It's just on contributors shoulders. I'm not interested in this mode and not ready to try to implement this, since it's PITA, definitely.
@pohmelie Ah ok, do you know of anyone in specific working on this? I'd like to help get this feature added.
@antonio-hickey AFAIK nobody even tried. So you can be the first.
Hi.
I added a quick implementation of explicit TLS (requires Python >= 3.11): 86a6a8c30c22e6611bed6cd3722de56936fc5d48.
Pass in explicit_tls=True
for Client()
or Client.context()
and the connection will automatically be upgraded prior to login. Alternatively, call client.upgrade_to_tls()
at any point to enable TLS. The data channel is automatically TLS encrypted if the command channel has been upgraded. Downgrading back to clear text with CCC or REIN commands is not supported.
For backwards compatibility, passing ssl=True
or ssl=yourcontext
will continue to do implicit TLS. If you specify both explicit_tls=True
and ssl=yourcontext
, the TLS upgrade will use your context (specifying ssl=True
or leaving it None
will use the default context).
Use default SSL context:
aioftp.Client("localhost", explicit_tls=True)
Use custom SSL context to bypass self-signed cert errors:
import ssl
sslcontext = ssl.create_default_context()
sslcontext.check_hostname = False
sslcontext.verify_mode = ssl.CERT_NONE
aioftp.Client("localhost", ssl=sslcontext, explicit_tls=True)
I haven't done much testing other than getting this to work within my own project, and I'm not sure how you'd like to handle the tests for it since you've disabled SSL tests, but I figure I'd start the discussion.
@sammichaels, sounds great! I think it is time to jump onto 3.11 and use recent fixes for ssl and your code to allow explicit switch. I will try to migrate project to modern technologies before this (pyproject.toml
, black
, ruff
, bump to 3.11+, etc.) and restore old implicit
tests. Then we can move forward with your approach. Thank you!
@pohmelie I'll continue updating the fork with my changes as there's much more to do, like error handling and SSL session reuse. Glad to hear you're considering adding explicit support!
@sammichaels I moved codebase to pyproject.toml
, black
, ruff
and pre-commit
tools. Minimal version bumped to 3.11
.
After #36 I read about FTPS, SFTP and FTP over SSH.
SFTP is SSH extension and we should ignore it
FTPS is extension of FTP (https://tools.ietf.org/html/rfc2228)
FTP over SSH is (as I realized) just tunneling and have problems(?) with data connections
[x] implicit ftps mode (#81)
[ ] explicit ftps mode