nodemcu / nodemcu-firmware

Lua based interactive firmware for ESP8266, ESP8285 and ESP32
https://nodemcu.readthedocs.io
MIT License
7.67k stars 3.13k forks source link

Simple FTP server #2343

Closed ghost closed 6 years ago

ghost commented 6 years ago

Missing feature

Simple FTP server

Justification

Some times need upload files withuot rs232 connection. FTP is best solution.

Workarounds

Just use lua for FTP Server Lua example:

wifi.setmode(wifi.SOFTAP)
wifi.ap.config({ssid="test",pwd="12345678"})
-- a simple ftp server
USER = "test"
PASS = "12345"
local file,net,pairs,print,string,table = file,net,pairs,print,string,table
data_fnc = nil
ftp_data = net.createServer(net.TCP, 180)
ftp_data:listen(20, function (s)
  if data_fnc then data_fnc(s) end
end)
ftp_srv = net.createServer(net.TCP, 180)
ftp_srv:listen(21, function(socket)
  local s = 0
  local cwd = "/"
  local buf = ""
  local t = 0
  socket:on("receive", function(c, d)
    a = {}
    for i in string.gmatch(d, "([^ \r\n]+)") do
      table.insert(a,i)
    end
    if(a[1] == nil or a[1] == "")then return end
    if(s == 0 and a[1] == "USER")then
      if(a[2] ~= USER)then
        return c:send("530 user not found\r\n")
      end
      s = 1
      return c:send("331 OK. Password required\r\n")
    end
    if(s == 1 and a[1] == "PASS")then
      if(a[2] ~= PASS)then
        return c:send("530 \r\n")
      end
      s = 2
      return c:send("230 OK.\r\n")
    end
    if(a[1] == "CDUP")then
      return c:send("250 OK. Current directory is "..cwd.."\r\n")
    end
    if(a[1] == "CWD")then
      if(a[2] == ".")then
        return c:send("257 \""..cwd.."\" is your current directory\r\n")
      end
      cwd = a[2]
      return c:send("250 OK. Current directory is "..cwd.."\r\n")
    end
    if(a[1] == "PWD")then
      return c:send("257 \""..cwd.."\" is your current directory\r\n")
    end
    if(a[1] == "TYPE")then
      if(a[2] == "A")then
        t = 0
        return c:send("200 TYPE is now ASII\r\n")
      end
      if(a[2] == "I")then
        t = 1
        return c:send("200 TYPE is now 8-bit binary\r\n")
      end
      return c:send("504 Unknown TYPE")
    end
    if(a[1] == "MODE")then
      if(a[2] ~= "S")then
        return c:send("504 Only S(tream) is suported\r\n")
      end
      return c:send("200 S OK\r\n")
    end
    if(a[1] == "PASV")then
      local _,ip = socket:getaddr()
      local _,_,i1,i2,i3,i4 = string.find(ip,"(%d+).(%d+).(%d+).(%d+)")
      return c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..",0,20).\r\n")
    end
    if(a[1] == "LIST" or a[1] == "NLST")then
      c:send("150 Accepted data connection\r\n")
      data_fnc = function(sd)
        local l = file.list();
        for k,v in pairs(l) do
          if(a[1] == "NLST")then
            sd:send(k.."\r\n")
          else
            sd:send("+r,s"..v..",\t"..k.."\r\n")
          end
        end
        sd:close()
        data_fnc = nil
        c:send("226 Transfer complete.\r\n")
      end
      return
    end
    if(a[1] == "RETR")then
      f = file.open(a[2]:gsub("%/",""),"r")
      if(f == nil)then
        return c:send("550 File "..a[2].." not found\r\n")
      end
      c:send("150 Accepted data connection\r\n")
      data_fnc = function(sd)
        local b=f:read(1024)
        sd:on("sent", function(cd)
          if b then
            sd:send(b)
            b=f:read(1024)
          else
            sd:close()
            f:close()
            data_fnc = nil
            c:send("226 Transfer complete.\r\n")
          end
        end)
        if b then
          sd:send(b)
          b=f:read(1024)
        end
      end
      return
    end
    if(a[1] == "STOR")then
      f = file.open(a[2]:gsub("%/",""),"w")
      if(f == nil)then
        return c:send("451 Can't open/create "..a[2].."\r\n")
      end
      c:send("150 Accepted data connection\r\n")
      data_fnc = function(sd)
        sd:on("receive", function(cd, dd)
          f:write(dd)
        end)
        socket:on("disconnection", function(c)
          f:close()
          data_fnc = nil
        end)
        c:send("226 Transfer complete.\r\n")
      end
      return
    end
    if(a[1] == "RNFR")then
      buf = a[2]
      return c:send("350 RNFR accepted\r\n")
    end
    if(a[1] == "RNTO" and buf ~= "")then
      file.rename(buf, a[2])
      buf = ""
      return c:send("250 File renamed\r\n")
    end
    if(a[1] == "DELE")then
      if(a[2] == nil or a[2] == "")then
        return c:send("501 No file name\r\n")
      end
      file.remove(a[2]:gsub("%/",""))
      return c:send("250 Deleted "..a[2].."\r\n")
    end
    if(a[1] == "NOOP")then
      return c:send("200 OK\r\n")
    end
    if(a[1] == "QUIT")then
      return c:send("221 Goodbye\r\n",function (s) s:close() end)
    end
    c:send("500 Unknown error\r\n")
  end)
  socket:send("220--- Welcome to FTP for ESP8266/ESP32 ---\r\n220---   By NeiroN   ---\r\n220 --   Version 1.4   --\r\n");
end)
FrankX0 commented 6 years ago

Nice! To get it working with FileZilla I had to make some changes though. When the ftp client sends a PASV command, the server should respond with the actual port used by the client, and start listening on that port. So I changed your code:

    if(a[1] == "PASV")then
        local _,_,i1,i2,i3,i4 = string.find(IP,"(%d+).(%d+).(%d+).(%d+)")
        c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..",0,20).\r\n")
        return
    end

into

        if(a[1] == "PASV")then
            local port, _ = c:getpeer()
            local _,_,i1,i2,i3,i4 = string.find(IP,"(%d+).(%d+).(%d+).(%d+)")
            c:send("227 Entering Passive Mode ("..i1..","..i2..","..i3..","..i4..","..(port / 256)..","..(port % 256)..").\r\n")
            if ftp_data ~= nil then
                ftp_data:close()
            end
            ftp_data = net.createServer(net.TCP, 180)
            ftp_data:listen(port, function (s)
                if data_fnc then
                    data_fnc(s)
                end
            end)
            return
        end

Not sure yet if this is the best way (there seems to be a memory leak this way), but now I can transfer files both ways with FileZilla. Any further improvements are welcome! Please note that sending multiple consecutive sends at the end of your code is not guaranteed to work. See the NodeMCU Documentation

TerryE commented 6 years ago

A general :+1: but could do with polishing, E.g

I've not pneumonia and on antibiotics, so don't really have the stamina to do this justice. Will pick itup in the next day or so. But this needs tidying, reformatting as a Lua_examples PR. Good job! :smile:

ghost commented 6 years ago

In a testing - i am try open port on every LIST,STOR,RETR command, but it produse memory leak after 1...3 cals and crash. It is no needs to start listen every PASV because port do not changes - port listens every time but pocess connectin if function exists.

P.S. Update code in first comment

devsaurus commented 6 years ago

I looped the sequence of

  1. log in
  2. quit

and memory is leaking already there.

In the socket:on("receive", function callback there are some references to socket which creates an unnecessary upval reference, potentially leaking memory. I recommend you replace these with c.

Furthermore, this line closes the socket right during sending:

return c:send("221 Goodbye\r\n"), socket:close()

It should be

return c:send("221 Goodbye\r\n", function (sock) sock:close() end)

With these changes I don't see memory leaks anymore for the login/quit sequence.

marcelstoer commented 6 years ago

Guys, it's a bit hard to track the code you send back and forth in comments here. @NeiroNx could you please create a PR for https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_modules to allow people to comment on actual code lines.

ghost commented 6 years ago

OK making pull.

drawkula commented 6 years ago

Firefox, LFTP and Nautilus seem to be happy now with ftpserver.py. MidnightCommander coughs while reading the directory and fails. ...but even without MC liking it, it is very useful!

Many thanks!


Edit @ 20180428-1300-GMT

Midnight-Commander now is able to get the directory list but cannot access the files:

20180428-124015-gmt

...but I have lost track of which of the suggested changes I have applied manually, so this comment probably is worthless... :-(

I'll retry Midnight-Commander when the next changes show up in the PR.

TerryE commented 6 years ago

BTW guys, I don't know how active data transfer mode ever worked. With this, the client issues a PORT command and the server (the ESP) makes an outbound connection from port 21 to the client at the designated port to do the data transfer. The net module (specifically net.socket:connect()) doesn't allow the Lua application to specify the local port for an outbound connection, so I am not sure that this will work as most clients will check the incoming port number to make sure than the connect request is from port 21.

I need to check the espconn source and LwIP to see if we can even make the change to allow this with the ESP stack. Failing this, it's PASV only, but luckily most clients default to PASV so that they can work through ADSL routers.

PS. Thanks to #1836, the SO_REUSE option is enabled which allow the connection request to set the pcb->local_port if pcb->so_options |= SOF_REUSEADDR. so the net.c patch is in principle doable, but the easiest thing in the short term is always to use passive transfer.

TerryE commented 6 years ago

OK, I've got my server working stably against a couple of FTP clients. See this gist for the source.

@drawkula @devsaurus could you try this out to see if you find any issues. At the moment, it dumps a lot of debug to the UART0, but you can either comment out the print statement in the debug routine or do a global edit debug( -> -- debug( to turn this off.

drawkula commented 6 years ago

I only get ls -1 like directory listings with the new version (using lftp). Midnight-Commander and Firefox don't work at all. Currently I only have a non-LFS system at hand.

TerryE commented 6 years ago

@drawkula If you are using a non-LFS build then comment out the debug and node.compile it. I'll take a look at the lftp etc and some C code examples which correctly give side info. Thanks

TerryE commented 6 years ago

@drawkula, The main issue seemed to be that many FTP servers would only correctly parse a unix ls -l style listing if the server returns a Unix type. Firefox also prefixes the filename with a path so when I strip off the leading / then this works fine as well. I've updated the gist version.

devsaurus commented 6 years ago

@drawkula @devsaurus could you try this out to see if you find any issues

I can't get this beast to run on non-LFS dev. Always barks at me even though I removed all debug:

> node.compile("ftpserver.lua")
E:M 136
stdin:1: not enough memory
stack traceback:
    [C]: in function 'compile'
    stdin:1: in main chunk

Will need to set up an LFS image I guess.

FrankX0 commented 6 years ago

@devsaurus or use an integer build. Works fine so far with FileZilla and in Windows explorer.

TerryE commented 6 years ago

Or use luac.cross and esplorer to bootstrap up the first lc file. The problem is the dynamic of compiling large files especially with the 16 byte TValue builds. I checked that it could compile on my build, but I use 12 byte TValues.

As I said LFS spoils you. I have this, telnet, my provisioning system and a bunch of utilities loaded at boot, and I still have 45 Kb RAM free.

I did spend about an hour during testing working out why my updating LFS seemed to have stopped. I even used esptool to read the LFS back to a file and did a diff - it was only then that the penny dropped, and I had drag and dropped a copy of the LC file to SPIFFS and the require was picking this up in preference. Durrhhh! - idiot.

If you do get a version with debug running, then have a look at how uploading large files works - no need to set max upload rates to prevent ESP overrun.

TerryE commented 6 years ago

Another power trick when trying to make sure that you are GCing all resources is to enumerate debug.getregistry() and look for any userdata or function resources listed. The getmeta of the userdata can be enumerated to give you it's methods and this will tell you what it is. You can also just call the function and the error (unless you've striped the error info with node.compile) will tell you which function it is, e.g. debug.getregistry()[2]()

devsaurus commented 6 years ago

Got it working in LFS against standard Linux ftp and ncftp clients.

Just a corner case, but listing an empty SPIFFS causes a panic in line 469 no data to send :wink:

TerryE commented 6 years ago

OK, new version uploaded. I've tried this against FileZilla, Chrome, Firefox, ftp, lftp, and ES File explorer on Android.

I've just hammered an FTP instance with all of the above, sometimes multiple concurrently, and when the last was closed, the heap was 41,536 on my LFS build, and following a FTP.close() to close server this rose to 43,616

drawkula commented 6 years ago

lftp and Firefox are happy with the latest version (on a non-LFS build, lua.cross compiled on the PC).

Midnight Commander now connects, but shows...

grafik

After changing the tab before the filename to a space:

grafik

Accessing the files still does not work.

TerryE commented 6 years ago

mc variously prefixes file names with /, /./ and ././ which is why the file operations were failing. I've added logic to strip these off, but not the full compression of arbitrary paths to canonical form. At least it now works (and the gist updated).

I can understand a command line FTP utility, but why use 1970s-style CUA? IBM 3270s don't exist anymore. Try sudo apt-get install filezilla :laughing:

PS. I've added an extra debug swtich to the FTP open and createServer methods, so you only get debug output to the uart if this is true.

HHHartmann commented 6 years ago

tested on Win10 and early 2.1.0 int build with 72 files in SPIFFS. compile works well on a freshly booted device. Testing with windows explorer. The LIST command gave me an out of memory exception. Locking at the code I saw that the whole response is first prepared in memory and then sent. I altered the code to prepare only the next chunk and only hold the position of the last file processed.

Here is the new LIST section:

''' if cmd == "LIST" or cmd == "NLST" then -- There are local skip, pattern, user = 0, '.', FTP.user

arg = arg:gsub('^-[a-z]* *', '') -- ignore any Unix style command parameters
arg = arg:gsub('^/','')  -- ignore any leading /
if arg == '' or arg == '.' then  -- use a "." which matches any non-blank filename
  pattern = "."
else -- replace "*" by [^/%.]* that is any string not including / or .
  pattern = arg:gsub('*','[^/%%.]*')
end

function cxt.getData() -- upval: skip, pattern, user (, table)
  local list, listSize, count = {}, 0, 0
  debug("skip: %s", skip)
  for k,v in pairs(file.list()) do
    if count >= skip then
      if k:match(pattern) then
        local line = (cmd == "LIST") and    
          ("-rw-r--r-- 1 %s %s %6u Jan  1 00:00 %s\r\n"):format(user, user, v, k) or
          (k.."\r\n")
        -- Rebatch LS lines into ~1024 packed for efficient Xfer 
        if listSize + #line > 1024 then
          debug("returning %i lines", #list)
          return table.concat(list)
        end
        list[#list+1] = line
        listSize = listSize + #line
        skip = skip + 1
      end
    end
    count = count + 1
  end

  debug("returning %i lines", #list)
  return #list > 0 and table.concat(list) or nil
end

'''

Runtime is not as good, but peak memory consumption is in my case (72 files) around 4K less.

To have more consistency, loading the files list could also be moved outside cxt.getData.

While debuging this I also found, that the problem in the original code might also have been a missing '' listSize = listSize + #line

I also noticed, that files containing '/' can be downloaded in Windows Explorer without problem. It just creates the directory. Uploading fails at the MKD command. Would be really great to emulate directories somehow.

drawkula commented 6 years ago

I can understand a command line FTP utility, but why use 1970s-style CUA? IBM 3270s don't exist anymore. Try sudo apt-get install filezilla

No, thanks! I do not like GUIs!

:laughing:

:-1:

Was this comment really neccesary?

devsaurus commented 6 years ago

Uploading fails at the MKD command. Would be really great to emulate directories somehow.

That would be tricky, I assume. And should be done in the platform/vfs layer rather than on the application layer as fatfs supports directories out of the box.

Note that the ftpserver ignores fatfs at the moment and assumes there's only spiffs. It effectively chroots to /FLASH.

TerryE commented 6 years ago

Was this comment really necessary?

No, it was supposed to be a mild tease in fun. I don't have an objection to anyone using whatever they want. However you gave me a bug without a solution so I had to download and use mc to diagnose this, and I personally dislike this interface intensely. À chacun son goût. My apologies, if I've given offense..

My issue here is that we can't implement a full FTP server with canonical filename reduction, because I am trying to keep the code size down so that non LFS users can still use the server. If an additional client has foibles like prefixing file names with /-/ which depends on reduction to canonical form, then this is extra functionality to implement and to test.

Would be really great to emulate directories somehow

As far as the issue of directories and SPIFFS, the size and structure of the FS simply doesn't merit a hierarchical implementation; however SPIFFS treats / as another filename character and so there is nothing to stop you using xyz/ as a prefix and in effect a namespace within SPIFFS. We could therefore emulate directories within the FTP server, but again some clients do things like expect directories to exist and to create them so we would need extra code to spoof them in to work within their navigation rules (e.g. using the zero-length file xyz/ to mark a pseudo-directory). I just think that this is going to be hard to implement within the size constraints of in a non-LFS version. So my instinct is to defer this discussion for now.

Looking at my own post LFS style, the number of files n SPIFFS has imploded. This is for two reasons:

@HHHartmann Gregor, I'll have a look at your code, but thinking about this, the send batching is a nice to have. With FS_OBJ_NAME_LEN set at 31, the maximum line length is 46+31+2 bytes = 79 bytes, and the maximum packet size is 1460 or whatever so it would just be easier to dump the size test and output the listing in batches of 10 lines doing the formatting in the getData() routine as you suggest.
The main overhead from a RAM viewpoint is actually the file.list() creating the 72 filename strings and the 72 (actually 128 entry) hashed table, instead of it being an iterator (like pairs) which would have minimal RAM footprint. (However making this change would have backwards compatibility issues.) Big tables cause the Lua RTL to choke because of their RAM footprint.

However I would also prefer to do the output in sorted order which adds an extra indexed array and indexed arrays take up a lot less memory. So I tend to think of the constraints of using large numbers of files as just that: an intrinsic resource constraint of the ESP8266 that we should spend too much time on trying to code around.

drawkula commented 6 years ago

Was this comment really necessary?

No, it was supposed to be a mild tease in fun.

Ok... humor is difficult to translate.

TerryE commented 6 years ago

@HHHartmann Gregor, I've taken your suggestion and tweaked it slightly. This drops the peak RAM usage by deferring the list formatting on a JiT basis to the getData routine. I've avoided doing the file.list() multiple times as this is an expensive operation and complicates processing. The extra array table for the filnames is quite cheap on RAM since all of the keys are already in the RAM strt, and the array form has small overheads (essentially a TValue * vector) plus the Table header.

The updated version is in the gist.

@marcelstoer, I will raise this as a PR after we've done the LFS PR.

marcelstoer commented 6 years ago

Terry, you'll then close this PR as yours will replace it, right?

HHHartmann commented 6 years ago

@TerryE Terry, thanks for taking the suggestion. Filtering for the file pattern as you do sure makes sense. From locking at your code I have two comments: After formatting the string to be sent you could release the file entry: '' fileSize[f] = nil The approach of sending batches of 10 is ok for the LIST command, but not so nice for NLST. But event for the LIST command it wastes about 23% of space in each packet, assuming that the filename length is evenly distributed around 15 characters.

Thinking about emulating directories I think that it would be possible to have a consintent enough experience for the user if we deduce directories with files in them from parsing the filenames. Newly created directories could be stored in an array accessible from all client connections. So there would be no Dir/ files or similar. There could be a separate module with methods for directory changing and creating and methods like ToSpiffsName and FromSpiffsName converting between the two worlds.

What do you think about that?

TerryE commented 6 years ago

Terry, you'll then close this PR as yours will replace it, right?

NeiroNx seems to have disengaged, so this would be the simplest thing to do.

After formatting the string to be sent you could release the file entry: fileSize[f] = nil

:laughing: I actually had that line in but accidentally deleted it when I removed some extra temporary debug. The main overhead of the table hash entries is the node array which has 2^n entries comprising 2 Tvalues and a next link which is 2*16+4+4 fill = 40bytes on a normal build, 32 bytes on the default LFS build and 24 on an integer build (though there is no reason why this needs to be double aligned so we could drop all of these by 4 bytes). So with 72 entries on a normal build this is 5,120 bytes and only reallocated to 2,560 bytes after 8 entries have been deleted.

If you were to optimise this server for non-LFS use, then you'd probably use overlays to significantly reduce the code loaded into RAM. If you've only got 20Kb RAM for variables after you've loaded the code as a single module, then chunks of 5Kb can be a resourcing problem.

As to emulating hierarchical FS on SPIFFS we have this issue that different FTP clients use a range of tactics. Some will index a directory e.g. CWD fred followed by LIST. Others might check the parent directory for fred existing, so we would need logic to:

So as I said, this logic is all getting very complicated, and you might be the only user that wants this. And I reiterate, with LFS about the only lua file that you have in SPIFFS is init.lua -- except perhaps when you are debugging a new module.

HHHartmann commented 6 years ago

Ah ok, I see. No luck in getting this feature. I got the folding of fred/ into one directory entry already done, Maybe I will continue for my own benefit once the "offichial" version has settled down.

Currently I have about as many non lua/lc files in SPIFFS as lua/lc files if not more. I like the ESP serving its own frontend, so I have http, js and json files sitting around, Other users of webservers would also like it.

But still great to have a working ftp server up and running. Downloading and deleting of "/" files works, so a script to rename say "fred#flintstone" to "fred/flintstone" would also work. Or entirely group files by '#' instead of '/'.

TerryE commented 6 years ago

I like the ESP serving its own frontend, so I have http, js and json files sitting around, Other users of webservers would also like it.

When code was executed from RAM, it made sense putting such smaller RO text resources as HTTP, JS, CSS each in their own file, but with LFS, why not create a Lua resource module on your host as part of the LFS build script, and then the resources are directly addressable from your code, e.g. LFS.resource("HTTP01"). OK, it still makes sense leaving larger binary resources such as PNG and JPEGs in their own files, but you shouldn't have dozens of these in a typical application.

topnotchrally commented 6 years ago

I have this code working on my ESP8255. It will connect and transfer a file to my Mac but will not do so to my Raspberry Pi, with ProFTPD server. I can connect/transfer to the RPi using Filezilla.

Can someone please help. I wish to FTP from the ESP to the RPi. Here is what I see on the ESP

WiFi connected; IP address: 192.168.2.100 WiFi mode: WIFI_STA Status: WL_CONNECTED Ready. Press d, u or r SPIFFS opened Command connected 220 ProFTPD 1.3.5 Server (Debian) [::ffff:192.168.2.148] Send USER 331 Password required for pi Send PASSWORD 230 User pi logged in Send SYST 215 UNIX Type: L8 Send Type I 200 Type set to I Send PASV 227 Entering Passive Mode (192,168,2,148,176,83). Data port: 45139 Data connection failed FTP FAIL

Thanks.

TerryE commented 6 years ago

I will take a look at this in a couple of days time.

topnotchrally commented 6 years ago

I would appreciate some help with this. I have dabbled a bit to diagnose the problem but can't figure out why the data connection fails.

TerryE commented 6 years ago

@topnotchrally I am a little confused as to what you are doing here. This example is an FTP server for ESP8266. ProFTPD is an FTP server for Linux systems, etc. FPT is a transfer protocol between a client and a server, not between 2 servers. You need to use an FTP client on your RPi or an FTP client on the ESP and this is a server, not a client. Different beasts.

topnotchrally commented 6 years ago

Sorry, my mistake I accidentally put this in the wrong place. I am using the FTP Client code for ESP8266.

This is what I am using. https://github.com/esp8266/Arduino/issues/1183

marcelstoer commented 6 years ago

Superseded by #2417 -> closing