Pyhtml sample code not working

[krishnak]

It complains about

    Global var "TestVar" : <span class="label">{{ TestVar }}</span><br />

TestVar not declared - I have tried setting TestVar in the main but it still complains. Can you provide a sample code for main

[jczic]

Hi @krishnak, You must set the global variable in your python code (see main.html example) like that :

pyhtmlMod.SetGlobalVar('TestVar', 12345)

[krishnak]

Thank you for a quick feedback, While this is not directly related to this issue, I hope you can throw me some light on what needs to be done. I was trying to write a Captive Portal using your webserver on ESP32. I have re-built micropython with your WebServer as a module, your webserver works great !!. All good so far. I then came across this project which has already implemented captive portal using your code https://github.com/george-hawkins/micropython-wifi-setup

I thought of reusing his code rather than reinventing the wheel, however he seems to have rewritten the /xasync_sockets.py HttpRequests etc.

My use case is slightly different than a mere Captive Portal, I want clients to access ESP32 via both STA as well as from AP

So can I just run two instances of the Webserver or is there a better optimised way to achieve this?

[jczic]

Yes, it is possible. However you should not use "StartManaged()" in this case, but rather "StartInPool()". There is already an example on the GIT page of MicroWebSrv2 with the title "Example to start a dual http/https web server". This example shows how to start 2 servers (http & https) in the same event POOL via "XAsyncSocketsPool". So you can start 2 HTTP servers on 2 different ports.

[krishnak]

I tried the following code snippet

`
wlan_ap.config(essid=localap_ssid, password=ap_password, authmode=ap_authmode)
xasPool  = XAsyncSocketsPool()
mws1=MicroWebSrv2()
mws1.SetEmbeddedConfig()
mws1.BindAddress=(accesspoint_addr,80)
mws1.RootPath = 'www'
mws1.StartInPool(xasPool)

print('Connect to WiFi ssid ' + localap_ssid + ', default password: ' + ap_password)
print('Listening on:', accesspoint_addr)
wlan = WiFiProvisioning.get_connection(wlan_sta)
if wlan is not None: 
    print("Connected to Router.....")
    print("Starting webserver on IP ",wlan.ifconfig()[0])
    """
    mws2=MicroWebSrv2()
    mws2.SetEmbeddedConfig()
    mws2.BindAddress=(station.ifconfig()[0],80)
    mws2.RootPath = 'www'
    mws2._slotsCount = 4
    mws2.StartInPool(xasPool)
    xasPool.AsyncWaitEvents(threadsCount=1)
    """
else:
    print("Could not initialize the network connection.")

xasPool.AsyncWaitEvents(threadsCount=1)

`

`MPY: soft reboot

   ---------------------------
   - Python pkg MicroWebSrv2 -
   -      version 2.0.6      -
   -     by JC`zic & HC2     -
   ---------------------------

MWS2-INFO> Server listening on 192.168.4.1:80. Connect to WiFi ssid SwitchBoard58bf259fc060, default password: 12345678 Listening on: 192.168.4.1 exception [Errno 2] ENOENT Could not initialize the network connection. MicroPython v1.19.1-dirty on 2022-08-30; ESP32 module with ESP32 Type "help()" for more information.

MWS2-DEBUG> From 192.168.4.2:44978 GET / >> [403] Forbidden MWS2-DEBUG> From 192.168.4.2:44978 GET / >> [400] Bad Request Unhandled exception in thread started by Traceback (most recent call last): File "MicroWebSrv2/libs/XAsyncSockets.py", line 131, in _processWaitEvents File "MicroWebSrv2/libs/XAsyncSockets.py", line 586, in OnReadyForReading XAsyncTCPClientException: Error when handling the "OnDataRecv" event : AsyncSendData : "data" is incorrect. `

I started the server only on the AP as the WiFi settings for station mode is purposefully not configured. The server starts up. I get a forbidden error and then the server throws the exception as above.

Please advise.

[jczic]

@krishnak, could you check "station.ifconfig()[0]". wlan is not None, okay, but probably not connected at this moment... Also, where the message "Could not initialize the network connection." comes from?

[krishnak]

That message is my debug statement from a function. I have enabled the other interface as well. i.e now listening on AP as well as STA , I get the exeption here as well. The only difference is the page gets served in STA, in AP it says forbidden 403. It appears that the server on AP is not having the same rootpath as the one on STA. Both the servers when they get a HTTP request throw the exception and the ESP32 reboots due to WDT.

` MPY: soft reboot

   ---------------------------
   - Python pkg MicroWebSrv2 -
   -      version 2.0.6      -
   -     by JC`zic & HC2     -
   ---------------------------

MWS2-INFO> Server listening on 192.168.4.1:80. ssid: Sundara chan: 5 rssi: -68 authmode: WPA/WPA2-PSK Trying to connect to Sundara... ............................................ Connected. Network config: ('192.168.1.12', '255.255.255.0', '192.168.1.1', '8.8.8.8') Connected to Router..... Starting webserver on IP 192.168.1.12 MWS2-INFO> Server listening on 192.168.1.12:80. Connect to WiFi ssid SwitchBoard58bf259fc060, default password: 12345678 Listening on: 192.168.4.1 MicroPython v1.19.1-dirty on 2022-08-30; ESP32 module with ESP32 Type "help()" for more information.

MWS2-DEBUG> From 192.168.1.11:47468 GET /favicon.ico >> [404] Not Found MWS2-DEBUG> From 192.168.1.11:47468 GET /favicon.ico >> [400] Bad Request Unhandled exception in thread started by Traceback (most recent call last): File "MicroWebSrv2/libs/XAsyncSockets.py", line 131, in _processWaitEvents File "MicroWebSrv2/libs/XAsyncSockets.py", line 586, in OnReadyForReading XAsyncTCPClientException: Error when handling the "OnDataRecv" event : AsyncSendData : "data" is incorrect. `

import WiFiProvisioning
from time import sleep
from MicroWebSrv2 import *
from MicroWebSrv2.libs import *
import machine
import network
import ubinascii
import config

led = machine.Pin(2, machine.Pin.OUT)

config.wlan_sta.active(True)
config.wlan_ap.active(True)

wlan_mac = ubinascii.hexlify(config.wlan_sta.config('mac')).decode()
localap_ssid = config.ap_ssid+str(wlan_mac)

config.wlan_ap.config(essid=localap_ssid, password=config.ap_password, authmode=config.ap_authmode)
xasPool  = XAsyncSocketsPool()
mws1=MicroWebSrv2()
mws1.SetEmbeddedConfig()
mws1.BindAddress=(config.accesspoint_addr,80)
mws1.RootPath = 'www'
mws1._slotsCount = 4
mws1.StartInPool(xasPool)

wlan = WiFiProvisioning.get_connection()
if wlan is not None: 
    print("Connected to Router.....")
    print("Starting webserver on IP ",wlan.ifconfig()[0])

    mws2=MicroWebSrv2()
    mws2.SetEmbeddedConfig()
    mws2.BindAddress=(config.wlan_sta.ifconfig()[0],80)
    mws2.RootPath = 'www'
    mws2._slotsCount = 4
    mws2.StartInPool(xasPool)
    xasPool.AsyncWaitEvents(threadsCount=1)

else:
    print("Could not initialize the network connection to router.")

print('Connect to WiFi ssid ' + localap_ssid + ', default password: ' + config.ap_password)
print('Listening on:comment', config.accesspoint_addr)

xasPool.AsyncWaitEvents(threadsCount=5)

[krishnak]

Update: I had xasPool.AsyncWaitEvents(threadsCount=1) in the else block and also xasPool.AsyncWaitEvents(threadsCount=5) at the end.

I have removed xasPool.AsyncWaitEvents(threadsCount=1)

Now the server is stable on STA - no exceptions.page gets served normally as it does on the managed server.

However the AP server still throws an exception

[krishnak]

Update 2: I have figured out the cause you need to give the solution.

The exceptions are related to URI/files not being found by the server.

The server on STA which is serving - crashes without exception (no longer accessible) if I try to access a non existent file.

As the server on Access point is not serving any files (i.e not finding the files) it is throwing exceptions all the time.

So please check whether you can reproduce this behaviour and decide whether its a bug.

[jczic]

I've this code in httpResponse line 355 :

        try :
            size = stat(filename)[6]
        except :
            self.ReturnNotFound()
            return

A found file is true if the size exists. May be your version of python returns 0 or None but but without causing an exception?

[krishnak]

httpResponse.zip

Attached is the httpresponse.zip which is built

Even if that is the case, why does the server running on AP not find the same file present in the root directory which the server on STA finds it.

[jczic]

Hmm, since you managed to make it work on several threads, there may be an inter-thread blocking issue... I don't see at the moment.

[jczic]

You can try to return a specific page on a "not found" page. But in my opinion it will crash out before the answer.

[krishnak]

I just found out that the browser is showing a cached page for the request to STA. I have shifted to wget which shows connection reset by peer for both STA as well as AP. I think there is some relation to number of threads and slot counts, I have reduced slot counts to 1 and it no longer works. What exactly does the slot count affect? I will try to install the firmware on another ESP32 tomorrow to see whether this is a hardware issue.

(--2022-08-31 00:02:29--  (try: 6)  http://192.168.1.12/
Connecting to 192.168.1.12:80... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.)

[krishnak]

I have rebuilt your code with some print statements on XAsyncSockets before it throws the exception in OnReadyForReading

It appears that there is empty line in the HTTP request and it throws error trying to decode it. see below actual line and decoded line

MWS2-INFO> Server listening on 192.168.4.1:80.
MicroPython v1.19.1-dirty on 2022-08-31; ESP32 module with ESP32
Type "help()" for more information.
>>> bytearray(b'GET /test.html HTTP/1.1')
GET /test.html HTTP/1.1
After decoding
bytearray(b'Host: 192.168.1.12')
Host: 192.168.1.12
After decoding
bytearray(b'Connection: keep-alive')
Connection: keep-alive
After decoding
bytearray(b'Cache-Control: max-age=0')
Cache-Control: max-age=0
After decoding
bytearray(b'DNT: 1')
DNT: 1
After decoding
bytearray(b'Upgrade-Insecure-Requests: 1')
Upgrade-Insecure-Requests: 1
After decoding
bytearray(b'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.105 Safari/537.36')
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.105 Safari/537.36
After decoding
bytearray(b'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9')
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
After decoding
bytearray(b'Accept-Encoding: gzip, deflate')
Accept-Encoding: gzip, deflate
After decoding
bytearray(b'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8')
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
After decoding
bytearray(b'')

The last byte array is the one throwing error

[jczic]

Ok @krishnak, this is a 'normal' raised exception in XAsyncSockets, not in OnReadyForReading but in AsyncSendData. Your error is : XAsyncTCPClientException: Error when handling the "OnDataRecv" event : AsyncSendData : "data" is incorrect. and the last exception "AsyncSendData : "data" is incorrect." is raised.

So, Here is the function :

    def AsyncSendData(self, data, onDataSent=None, onDataSentArg=None) :
        if self._socket :
            try :
                if bytes([data[0]]) :
                    if self._wrBufView :
                        self._wrBufView = memoryview(bytes(self._wrBufView) + data)
                    else :
                        self._wrBufView = memoryview(data)
                    self._onDataSent    = onDataSent
                    self._onDataSentArg = onDataSentArg
                    self._asyncSocketsPool.NotifyNextReadyForWriting(self, True)
                    self.OnReadyForWriting()
                    return True
            except :
                pass
            raise XAsyncTCPClientException('AsyncSendData : "data" is incorrect.')
        return False

The exception is catched here... Probably a memory problem when memoryview is called to proceed the received buffer.

If you want, you can set print here or catch the true exception :

except Exception as ex :
    print(ex)

instead of :

except :
    pass

[jczic]

Just @krishnak , you have probably reached the default thread stack size...

[krishnak]

Thanks for pointing the right function, I was assuming it was an error thrown at request side.

I have missed out one issue on my side when I reflashed a recompiled version the firmware, I forgot to create WWW folder in the flash. Hence it wasn't finding the files.

I have created this folder now, however the error still persists

Now I am running only one server on STA with Startin Pool

mws2._slotsCount = 8 # any value from 4 on wards make no difference
xasPool.AsyncWaitEvents(threadsCount=3)

However the behaviour is the same, if it finds the file it doesn't throw an exception, however it crashes before the full response is sent back- no further HTTP requests are possible after that. As you can see below there are two calls made to

rishnak@tapati:~$ telnet 192.168.1.12 80
Trying 192.168.1.12...
Connected to 192.168.1.12.
Escape character is '^]'.
GET /test.html HTTP/1.1

HTTP/1.1 200 OK
Server: MicroWebSrv2 by JC`zic
Connection: Close
Content-Type: text/html
Content-Length: 21
Cache-Control: public, max-age=31536000

The server side message is as follows, there is a 200 OK and then a 400 Bad request - I am not sure what is causing this , I think it is a threading issue (why the AsyncSendData is called twice?) when there is only one request., as the same hardware works in managed mode.

MWS2-INFO> Server listening on 192.168.1.12:80. MicroPython v1.19.1-dirty on 2022-08-31; ESP32 module with ESP32 Type "help()" for more information.

size inside HTTPresponse 21 MWS2-DEBUG> From 192.168.1.11:38988 GET /test.html >> [200] OK Inside AsyncSendData b'HTTP/1.1 200 OK\r\nServer: MicroWebSrv2 by JCzic\r\nConnection: Close\r\nContent-Type: text/html\r\nContent-Length: 21\r\nCache-Control: public, max-age=31536000\r\n\r\n' MWS2-DEBUG> From 192.168.1.11:38988 GET /test.html >> [400] Bad Request Inside AsyncSendData b'HTTP/1.1 400 Bad Request\r\nServer: MicroWebSrv2 by JCzic\r\nConnection: Close\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Length: 306\r\nCache-Control: public, max-age=31536000\r\n\r\n \n \n \n \n \n

MicroWebSrv2 - [400] Bad Request

\n Bad request syntax or unsupported method.\n \n \n '

[krishnak]

I have tested it again in ManagedPool it works as expected you can see the the server output below. I have found the issue it may not be memoryview,

The onDataSent function inside httpResponse.py hangs at xasCli.AsyncSendSendingBuffer around line 155 error.txt

if self._sendingBuf :
            print("2 if")
            if self._contentLength :
                print("inside if")
                self._xasCli.AsyncSendSendingBuffer( size       = len(self._sendingBuf),
                                                     onDataSent = self._onDataSent )
                if not self._stream :
                    print("inside if not")

                    self._sendingBuf = None

[krishnak]

Ok it actually is unable to get a lock here hence hangs

`

def _socketListAdd(self, socket, socketsList) :
        self._opLock.acquire()
        ok = (id(socket) in self._asyncSockets and socket not in socketsList)
        if ok :
            socketsList.append(socket)
        self._opLock.release()
        return ok`

[krishnak]

Actually its a simple fix, you need to put a thread.sleep before acquiring lock - there is a racing condition going on. I just added a print statement before acquire and that itself has given sufficient time for the threads to yield.

With this print statement, I am now able to run both servers. However there is still an uncaught exception causing the lock to be held by a thread which has died. This results in the requests from other threads waiting for the lock

no errors
Inside 5 if
Inside onDataSent
else block
Unhandled exception in thread started by <bound_method>
Traceback (most recent call last):
  File "MicroWebSrv2/libs/XAsyncSockets.py", line 135, in _processWaitEvents
  File "MicroWebSrv2/libs/XAsyncSockets.py", line 593, in OnReadyForReading
XAsyncTCPClientException: Error when handling the "OnDataRecv" event : AsyncSendData : "data" is incorrect.
waiting to acquire
1073656944

I think you can fix the code now.

[jczic]

Ok @krishnak, sleep() is really not the solution! The management of several threads is parallelism and the server also works in concurrent mode (the proof with only one thread or even none in the main thread).

If it works with several threads, it is because each thread has its memory while one thread does not reserve enough. Managed pool mode reserves 8x1024 bytes for its thread by default (which is a parameter that can be changed in the call to StartManaged).

Try to put these 2 lines in your main code, at the beginning :

from _thread import stack_size
stack_size(8*1024)

[krishnak]

I don't know python :) as deeply as you know. So I am happy to be enlightened. file:///home/krishnak/after.txt file:///home/krishnak/before.txt

What I observed. If 404 error occurs, in AsyncPool mode - a new thread started to complain of 400 Bad request. So I thought there is another request, but it appears that 404 error thread never completed its task and has triggered 400 which got hung up.

Your solution has fixed it :1st_place_medal:

I have tested both the servers running on AP as well STA, they both are now working see attached logs. Working for both scenarios file being present and file being absent. No Crashes.

I want to close this thread and thank you for resolving this issue.

[jczic]

Well, I just didn't think about it before because I haven't used it for a while.

And If you want, to create a captive portal during a Wi-Fi connection, you can use my little DNS server: https://github.com/jczic/MicroDNSSrv Set the default "404 not found" page to your root "/" : mws2.NotFoundURL = '/' # relative or absolute URL

[krishnak]

Will Do look in to your DNS server thanks.

I need to look in to this as well

https://github.com/glenn20/micropython-espnow-images

On Wed, 31 Aug 2022 at 16:16, Jean-Christophe Bos @.***> wrote:

Well, I just didn't think about it before because I haven't used it for a while. And If you want, to create a captive portal during a Wi-Fi connection, you can use my little DNS server: https://github.com/jczic/MicroDNSSrv (and set the default "404 not found" page to your root "/")

— Reply to this email directly, view it on GitHub https://github.com/jczic/MicroWebSrv2/issues/84#issuecomment-1232775377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTPN6NCFR4F6W47QODVBTV34ZYPANCNFSM6AAAAAAQAG42A4 . You are receiving this because you modified the open/close state.Message ID: @.***>

[krishnak]

When I set

mws2.NotFoundURL = '/'

mws2.NotFoundURL = '/notfound.html' # relative or absolute URL

The server crashes after first hit even for pages which are present, when I remove this line, everything works normally. Please check and advise.

[krishnak]

Also if

xasPool.AsyncWaitEvents(threadsCount=2)

if I increase this threadcount=3 - DNS Server doesn't start

Can you please explain what is the slot count and what is thread count

[krishnak]

Just an update on the not found url, it works in managed mode. On AsyncPool it is crashing. The not found "logic" in your code is spawning many more threads in Async mode than what is spawned in Managed mode when a file is not found. I am judging this mainly from the number of thread lock messages that gets printed for a not found URL.

in Managed mode the number of thread locks requested when a file is NOT FOUND is 10% of that of Async mode

[krishnak]

I think there is something specific happening with two socket operations involving the AP interface.

I am able to run your microDNS server on STA interface along with 2 webservers (one each on AP and STA under managed mode) - however when I run the microDNS server or for that matter any two threads involving sockets on AP, only one thread has the CPU access at a time, this probably is the cause for this whole issue

I have now tested this on 3 different hardwares to rule out hardware issue.,

[krishnak]

I have found out the cause for the problem, on my Ubuntu machine when the WiFi network changes to connect to the ESP32's WiFi AP the Hotspot app opens up, this is resulting in an infinite number of calls to - this makes everything crash. However if I keep the connectivity checking switched OFF then everything works but there is some thing to investigate with the async thread implementation, The DNS server doesn't work for the same reason. I have written a different DNS server with poller and it works along with the HTTPServer.

def OnReadyForWriting(self) :
        print("OnReadyForWriting")

OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting
OnReadyForWriting

[krishnak]

With the Ubuntu auto connectivity switched OFF, the Startinpool works with two servers - no crashing. The crash is due to the code spawning too many threads/sockets when there is a not found scenario. This needs fixing.