Closed ghost closed 6 years ago
Nice! To get it working with FileZilla I had to make some changes though. When the ftp client sends a PASV command, the server should respond with the actual port used by the client, and start listening on that port. So I changed your code:
if(a[1] == "PASV")then
local _,_,i1,i2,i3,i4 = string.find(IP,"(%d+).(%d+).(%d+).(%d+)")
c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..",0,20).\r\n")
return
end
into
if(a[1] == "PASV")then
local port, _ = c:getpeer()
local _,_,i1,i2,i3,i4 = string.find(IP,"(%d+).(%d+).(%d+).(%d+)")
c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..","..(port / 256)..","..(port % 256)..").\r\n")
if ftp_data ~= nil then
ftp_data:close()
end
ftp_data = net.createServer(net.TCP, 180)
ftp_data:listen(port, function (s)
if data_fnc then
data_fnc(s)
end
end)
return
end
Not sure yet if this is the best way (there seems to be a memory leak this way), but now I can transfer files both ways with FileZilla. Any further improvements are welcome! Please note that sending multiple consecutive sends at the end of your code is not guaranteed to work. See the NodeMCU Documentation
A general :+1: but could do with polishing, E.g
local file,net,pairs,print,string,table,wifi = file,net,pairs,print,string,table,wifi
return c:send("200 OK\r');
socket:send(
"220--- Welcome to FTP for ESP8266/ESP32 ---\r\n220--- By NeiroN ---\r\n220 -- Version 1.0 --\r\n"
);
I've not pneumonia and on antibiotics, so don't really have the stamina to do this justice. Will pick itup in the next day or so. But this needs tidying, reformatting as a Lua_examples PR. Good job! :smile:
In a testing - i am try open port on every LIST,STOR,RETR command, but it produse memory leak after 1...3 cals and crash. It is no needs to start listen every PASV because port do not changes - port listens every time but pocess connectin if function exists.
P.S. Update code in first comment
I looped the sequence of
and memory is leaking already there.
In the socket:on("receive", function
callback there are some references to socket
which creates an unnecessary upval reference, potentially leaking memory. I recommend you replace these with c
.
Furthermore, this line closes the socket right during sending:
return c:send("221 Goodbye\r\n"), socket:close()
It should be
return c:send("221 Goodbye\r\n", function (sock) sock:close() end)
With these changes I don't see memory leaks anymore for the login/quit sequence.
Guys, it's a bit hard to track the code you send back and forth in comments here. @NeiroNx could you please create a PR for https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_modules to allow people to comment on actual code lines.
OK making pull.
Firefox, LFTP and Nautilus seem to be happy now with ftpserver.py
.
MidnightCommander coughs while reading the directory and fails.
...but even without MC liking it, it is very useful!
Many thanks!
Edit @ 20180428-1300-GMT
Midnight-Commander now is able to get the directory list but cannot access the files:
...but I have lost track of which of the suggested changes I have applied manually, so this comment probably is worthless... :-(
I'll retry Midnight-Commander when the next changes show up in the PR.
BTW guys, I don't know how active data transfer mode ever worked. With this, the client issues a PORT command and the server (the ESP) makes an outbound connection from port 21 to the client at the designated port to do the data transfer. The net module (specifically net.socket:connect()
) doesn't allow the Lua application to specify the local port for an outbound connection, so I am not sure that this will work as most clients will check the incoming port number to make sure than the connect request is from port 21.
I need to check the espconn source and LwIP to see if we can even make the change to allow this with the ESP stack. Failing this, it's PASV only, but luckily most clients default to PASV so that they can work through ADSL routers.
PS. Thanks to #1836, the SO_REUSE
option is enabled which allow the connection request to set the pcb->local_port
if pcb->so_options |= SOF_REUSEADDR
. so the net.c
patch is in principle doable, but the easiest thing in the short term is always to use passive transfer.
OK, I've got my server working stably against a couple of FTP clients. See this gist for the source.
@drawkula @devsaurus could you try this out to see if you find any issues. At the moment, it dumps a lot of debug to the UART0, but you can either comment out the print
statement in the debug routine or do a global edit debug(
-> -- debug(
to turn this off.
put
operations, and used hold / unhold to prevent the net inbound packets overflowing the SPIFFS write rate. I only get ls -1
like directory listings with the new version (using lftp
).
Midnight-Commander and Firefox don't work at all.
Currently I only have a non-LFS system at hand.
@drawkula If you are using a non-LFS build then comment out the debug and node.compile it. I'll take a look at the lftp etc and some C code examples which correctly give side info. Thanks
@drawkula, The main issue seemed to be that many FTP servers would only correctly parse a unix ls -l
style listing if the server returns a Unix type. Firefox also prefixes the filename with a path so when I strip off the leading /
then this works fine as well. I've updated the gist version.
@drawkula @devsaurus could you try this out to see if you find any issues
I can't get this beast to run on non-LFS dev
. Always barks at me even though I removed all debug:
> node.compile("ftpserver.lua")
E:M 136
stdin:1: not enough memory
stack traceback:
[C]: in function 'compile'
stdin:1: in main chunk
Will need to set up an LFS image I guess.
@devsaurus or use an integer build. Works fine so far with FileZilla and in Windows explorer.
Or use luac.cross
and esplorer to bootstrap up the first lc file. The problem is the dynamic of compiling large files especially with the 16 byte TValue builds. I checked that it could compile on my build, but I use 12 byte TValues.
As I said LFS spoils you. I have this, telnet, my provisioning system and a bunch of utilities loaded at boot, and I still have 45 Kb RAM free.
I did spend about an hour during testing working out why my updating LFS seemed to have stopped. I even used esptool to read the LFS back to a file and did a diff - it was only then that the penny dropped, and I had drag and dropped a copy of the LC file to SPIFFS and the require was picking this up in preference. Durrhhh! - idiot.
If you do get a version with debug running, then have a look at how uploading large files works - no need to set max upload rates to prevent ESP overrun.
Another power trick when trying to make sure that you are GCing all resources is to enumerate debug.getregistry()
and look for any userdata or function resources listed. The getmeta of the userdata can be enumerated to give you it's methods and this will tell you what it is. You can also just call the function and the error (unless you've striped the error info with node.compile) will tell you which function it is, e.g. debug.getregistry()[2]()
Got it working in LFS against standard Linux ftp
and ncftp
clients.
Just a corner case, but listing an empty SPIFFS causes a panic in line 469 no data to send
:wink:
OK, new version uploaded. I've tried this against FileZilla, Chrome, Firefox, ftp, lftp, and ES File explorer on Android.
cwd
always returns an error unless the arg is / or . pwd
is always //
is ignored, and wildcards work but don't match /
or .
so for example *.lua
will list all Lua files.*.fredddszzs
will return no lines without crashing.I've just hammered an FTP instance with all of the above, sometimes multiple concurrently, and when the last was closed, the heap was 41,536 on my LFS build, and following a FTP.close()
to close server this rose to 43,616
lftp
and Firefox are happy with the latest version (on a non-LFS build, lua.cross
compiled on the PC).
Midnight Commander now connects, but shows...
After changing the tab before the filename to a space:
Accessing the files still does not work.
mc
variously prefixes file names with /
, /./
and ././
which is why the file operations were failing. I've added logic to strip these off, but not the full compression of arbitrary paths to canonical form. At least it now works (and the gist updated).
I can understand a command line FTP utility, but why use 1970s-style CUA? IBM 3270s don't exist anymore. Try sudo apt-get install filezilla
:laughing:
PS. I've added an extra debug swtich to the FTP open and createServer methods, so you only get debug output to the uart if this is true.
tested on Win10 and early 2.1.0 int build with 72 files in SPIFFS. compile works well on a freshly booted device. Testing with windows explorer. The LIST command gave me an out of memory exception. Locking at the code I saw that the whole response is first prepared in memory and then sent. I altered the code to prepare only the next chunk and only hold the position of the last file processed.
Here is the new LIST section:
''' if cmd == "LIST" or cmd == "NLST" then -- There are local skip, pattern, user = 0, '.', FTP.user
arg = arg:gsub('^-[a-z]* *', '') -- ignore any Unix style command parameters
arg = arg:gsub('^/','') -- ignore any leading /
if arg == '' or arg == '.' then -- use a "." which matches any non-blank filename
pattern = "."
else -- replace "*" by [^/%.]* that is any string not including / or .
pattern = arg:gsub('*','[^/%%.]*')
end
function cxt.getData() -- upval: skip, pattern, user (, table)
local list, listSize, count = {}, 0, 0
debug("skip: %s", skip)
for k,v in pairs(file.list()) do
if count >= skip then
if k:match(pattern) then
local line = (cmd == "LIST") and
("-rw-r--r-- 1 %s %s %6u Jan 1 00:00 %s\r\n"):format(user, user, v, k) or
(k.."\r\n")
-- Rebatch LS lines into ~1024 packed for efficient Xfer
if listSize + #line > 1024 then
debug("returning %i lines", #list)
return table.concat(list)
end
list[#list+1] = line
listSize = listSize + #line
skip = skip + 1
end
end
count = count + 1
end
debug("returning %i lines", #list)
return #list > 0 and table.concat(list) or nil
end
'''
Runtime is not as good, but peak memory consumption is in my case (72 files) around 4K less.
To have more consistency, loading the files list could also be moved outside cxt.getData.
While debuging this I also found, that the problem in the original code might also have been a missing '' listSize = listSize + #line
I also noticed, that files containing '/' can be downloaded in Windows Explorer without problem. It just creates the directory. Uploading fails at the MKD command. Would be really great to emulate directories somehow.
I can understand a command line FTP utility, but why use 1970s-style CUA? IBM 3270s don't exist anymore. Try sudo apt-get install filezilla
No, thanks! I do not like GUIs!
:laughing:
:-1:
Was this comment really neccesary?
Uploading fails at the MKD command. Would be really great to emulate directories somehow.
That would be tricky, I assume. And should be done in the platform/vfs layer rather than on the application layer as fatfs supports directories out of the box.
Note that the ftpserver ignores fatfs at the moment and assumes there's only spiffs. It effectively chroots to /FLASH
.
Was this comment really necessary?
No, it was supposed to be a mild tease in fun. I don't have an objection to anyone using whatever they want. However you gave me a bug without a solution so I had to download and use mc
to diagnose this, and I personally dislike this interface intensely. À chacun son goût. My apologies, if I've given offense..
My issue here is that we can't implement a full FTP server with canonical filename reduction, because I am trying to keep the code size down so that non LFS users can still use the server. If an additional client has foibles like prefixing file names with /-/
which depends on reduction to canonical form, then this is extra functionality to implement and to test.
Would be really great to emulate directories somehow
As far as the issue of directories and SPIFFS, the size and structure of the FS simply doesn't merit a hierarchical implementation; however SPIFFS treats /
as another filename character and so there is nothing to stop you using xyz/
as a prefix and in effect a namespace within SPIFFS. We could therefore emulate directories within the FTP server, but again some clients do things like expect directories to exist and to create them so we would need extra code to spoof them in to work within their navigation rules (e.g. using the zero-length file xyz/
to mark a pseudo-directory). I just think that this is going to be hard to implement within the size constraints of in a non-LFS version. So my instinct is to defer this discussion for now.
Looking at my own post LFS style, the number of files n SPIFFS has imploded. This is for two reasons:
I've Johny's suggestion of using the absolute address LFS is just far simpler. I have a script which uses luac.cross to recompile all files in an lfs
subdirectory on my host and esptool
directly to the LFS region and reboot the ESP in under 5sec, so the source files never touch SPIFFS.
Before LFS and using luac.cross
I used to keep my source files small -- say under 100 lines -- so a logical module might be implemented by 5 JiT-loaded files. Now I just put all of the module logic in a single file.
@HHHartmann Gregor, I'll have a look at your code, but thinking about this, the send batching is a nice to have. With FS_OBJ_NAME_LEN
set at 31, the maximum line length is 46+31+2 bytes = 79 bytes, and the maximum packet size is 1460 or whatever so it would just be easier to dump the size test and output the listing in batches of 10 lines doing the formatting in the getData()
routine as you suggest.
The main overhead from a RAM viewpoint is actually the file.list() creating the 72 filename strings and the 72 (actually 128 entry) hashed table, instead of it being an iterator (like pairs
) which would have minimal RAM footprint. (However making this change would have backwards compatibility issues.) Big tables cause the Lua RTL to choke because of their RAM footprint.
However I would also prefer to do the output in sorted order which adds an extra indexed array and indexed arrays take up a lot less memory. So I tend to think of the constraints of using large numbers of files as just that: an intrinsic resource constraint of the ESP8266 that we should spend too much time on trying to code around.
Was this comment really necessary?
No, it was supposed to be a mild tease in fun.
Ok... humor is difficult to translate.
@HHHartmann Gregor, I've taken your suggestion and tweaked it slightly. This drops the peak RAM usage by deferring the list formatting on a JiT basis to the getData routine. I've avoided doing the file.list() multiple times as this is an expensive operation and complicates processing. The extra array table for the filnames is quite cheap on RAM since all of the keys are already in the RAM strt, and the array form has small overheads (essentially a TValue *
vector) plus the Table
header.
The updated version is in the gist.
@marcelstoer, I will raise this as a PR after we've done the LFS PR.
Terry, you'll then close this PR as yours will replace it, right?
@TerryE Terry, thanks for taking the suggestion. Filtering for the file pattern as you do sure makes sense. From locking at your code I have two comments: After formatting the string to be sent you could release the file entry: '' fileSize[f] = nil The approach of sending batches of 10 is ok for the LIST command, but not so nice for NLST. But event for the LIST command it wastes about 23% of space in each packet, assuming that the filename length is evenly distributed around 15 characters.
Thinking about emulating directories I think that it would be possible to have a consintent enough experience for the user if we deduce directories with files in them from parsing the filenames. Newly created directories could be stored in an array accessible from all client connections. So there would be no Dir/ files or similar. There could be a separate module with methods for directory changing and creating and methods like ToSpiffsName and FromSpiffsName converting between the two worlds.
What do you think about that?
Terry, you'll then close this PR as yours will replace it, right?
NeiroNx seems to have disengaged, so this would be the simplest thing to do.
After formatting the string to be sent you could release the file entry:
fileSize[f] = nil
:laughing: I actually had that line in but accidentally deleted it when I removed some extra temporary debug. The main overhead of the table hash entries is the node
array which has 2^n entries comprising 2 Tvalue
s and a next link which is 2*16+4+4 fill = 40bytes on a normal build, 32 bytes on the default LFS build and 24 on an integer build (though there is no reason why this needs to be double aligned so we could drop all of these by 4 bytes). So with 72 entries on a normal build this is 5,120 bytes and only reallocated to 2,560 bytes after 8 entries have been deleted.
If you were to optimise this server for non-LFS use, then you'd probably use overlays to significantly reduce the code loaded into RAM. If you've only got 20Kb RAM for variables after you've loaded the code as a single module, then chunks of 5Kb can be a resourcing problem.
As to emulating hierarchical FS on SPIFFS we have this issue that different FTP clients use a range of tactics. Some will index a directory e.g. CWD fred
followed by LIST
. Others might check the parent directory for fred
existing, so we would need logic to:
fred/
with 0 size as a pseudo directory andfred/*
into a single entry fred/
unless of course the command was LIST fred/*
So as I said, this logic is all getting very complicated, and you might be the only user that wants this. And I reiterate, with LFS about the only lua
file that you have in SPIFFS is init.lua
-- except perhaps when you are debugging a new module.
Ah ok, I see. No luck in getting this feature. I got the folding of fred/ into one directory entry already done, Maybe I will continue for my own benefit once the "offichial" version has settled down.
Currently I have about as many non lua/lc files in SPIFFS as lua/lc files if not more. I like the ESP serving its own frontend, so I have http, js and json files sitting around, Other users of webservers would also like it.
But still great to have a working ftp server up and running. Downloading and deleting of "/" files works, so a script to rename say "fred#flintstone" to "fred/flintstone" would also work. Or entirely group files by '#' instead of '/'.
I like the ESP serving its own frontend, so I have http, js and json files sitting around, Other users of webservers would also like it.
When code was executed from RAM, it made sense putting such smaller RO text resources as HTTP, JS, CSS each in their own file, but with LFS, why not create a Lua resource module on your host as part of the LFS build script, and then the resources are directly addressable from your code, e.g. LFS.resource("HTTP01")
. OK, it still makes sense leaving larger binary resources such as PNG and JPEGs in their own files, but you shouldn't have dozens of these in a typical application.
I have this code working on my ESP8255. It will connect and transfer a file to my Mac but will not do so to my Raspberry Pi, with ProFTPD server. I can connect/transfer to the RPi using Filezilla.
Can someone please help. I wish to FTP from the ESP to the RPi. Here is what I see on the ESP
WiFi connected; IP address: 192.168.2.100 WiFi mode: WIFI_STA Status: WL_CONNECTED Ready. Press d, u or r SPIFFS opened Command connected 220 ProFTPD 1.3.5 Server (Debian) [::ffff:192.168.2.148] Send USER 331 Password required for pi Send PASSWORD 230 User pi logged in Send SYST 215 UNIX Type: L8 Send Type I 200 Type set to I Send PASV 227 Entering Passive Mode (192,168,2,148,176,83). Data port: 45139 Data connection failed FTP FAIL
Thanks.
I will take a look at this in a couple of days time.
I would appreciate some help with this. I have dabbled a bit to diagnose the problem but can't figure out why the data connection fails.
@topnotchrally I am a little confused as to what you are doing here. This example is an FTP server for ESP8266. ProFTPD is an FTP server for Linux systems, etc. FPT is a transfer protocol between a client and a server, not between 2 servers. You need to use an FTP client on your RPi or an FTP client on the ESP and this is a server, not a client. Different beasts.
Sorry, my mistake I accidentally put this in the wrong place. I am using the FTP Client code for ESP8266.
This is what I am using. https://github.com/esp8266/Arduino/issues/1183
Superseded by #2417 -> closing
Missing feature
Simple FTP server
Justification
Some times need upload files withuot rs232 connection. FTP is best solution.
Workarounds
Just use lua for FTP Server Lua example: