Module is imported partially

[MaxFalcone]

Describe the Bug

There is a module called PSLiteDB . Import-Module command imports module to Pode server. But some functions of module are absent.

Steps To Reproduce

Steps to reproduce the behavior:

1. Install-Module PSLiteDB
2. Start pode server as per script below

   
   [cmdletbinding()]
   param(
   [int] $port = 8090,
   [switch] $debugging
   )
   $PodeServerParams = @{"Quiet" = !$debugging; "DisableTermination" = !$debugging }

Import-Module Pode -Force #Does not work here

Start-PodeServer @PodeServerParams { Add-PodeEndpoint -Address localhost -Port $port -Protocol Http

Import-PodeModule -Name "PSLiteDB" #Does not work here either

Add-PodeRoute -Method Get -Path '/awake' -ScriptBlock {

Import-Module PSLiteDB -Force #Works here!!!

Get-Module PSLiteDB | out-string | Write-PodeTextResponse -StatusCode 200 Get-Command -Module PSLiteDB | out-string | Write-PodeTextResponse -StatusCode 200 } }

3. Open the route in browser. Check that `ConvertTo-LiteDbBSON` function is not available in the workspace of the route. It will be missing in the response.
![image](https://user-images.githubusercontent.com/72614595/147575170-680bec24-9d53-4320-a796-dc71dfc4477b.png)
4. Same happens if I put Import-Module inside the `Start-PodeServer`'s scriptblock or if I use `Import-PodeModule`.
5. But **it works** if I Import-Module right inside the `Add-PodeRoute`'s scriptblock.
![image](https://user-images.githubusercontent.com/72614595/147575770-9f092eec-6864-4b2a-a8cf-7082c1e74678.png)

### Expected Behavior
Function should be available if module imported before `Start-PodeServer`'s scriptblock.

### Platform
 - OS: Windows Server 2019
 - Browser: any
 - Versions:
   - Pode: 2.5.1
   - PowerShell: 7.2

### Additional info
Module is installed for all users in powershell 7.
![image](https://user-images.githubusercontent.com/72614595/147576196-0230a5ae-2d7f-459f-97f5-865ab9f532bc.png)

Also noticed that on the first screenshot (when module is imported globally) it shows version 0.0.1.0. I don't have such version installed.

[robinmalik]

Odd! I just installed PSLiteDB (never had it before) - scoped for all users, so into C:\Program Files\PowerShell\Modules and it works for me without needing an explicit import:

Code:

[cmdletbinding()]
param(
[int] $port = 8090,
[switch] $debugging
)
$PodeServerParams = @{"Quiet" = !$debugging; "DisableTermination" = !$debugging }

Start-PodeServer @PodeServerParams {
    Add-PodeEndpoint -Address localhost -Port $port -Protocol Http

    Add-PodeRoute -Method Get -Path '/awake' -ScriptBlock {
        Get-Module -Name PSLiteDB | out-string | Write-PodeTextResponse -StatusCode 200
        Get-Command -Module PSLiteDB | out-string | Write-PodeTextResponse -StatusCode 200
    }
}

Using Pode version 2.5.1.

Have you searched your system for PSLiteDB just to see if there's anything lingering around in other module locations? $env:psmodulepath -split ";" | sort is handy for seeing just where PS is pulling from.

[MaxFalcone]

Strange. Just created fresh server 2019. Installed powershell 7 and both Pode and PSLiteDB modules. If I open terminal window I can immediately run Pode commands with autocomplete (means module is automatically loaded), but PSLiteDB requires explicit Import-Module to run and to autocomplete. Same goes on my Windows 10 laptop. Is it possible that some kind of policy blocks it from autoloading?

[robinmalik]

My test was on my Windows 10 desktop. Repeating today from Terminal (and PS 7.2) as you can see autocomplete works, even before importing the module.

Historically autocomplete used to be managed by a CommandAnalysis cache populated on PS startup, but this has changed to ModuleAnalysisCache.

If you clear all ModuleCacheAnalysis files from $env:LOCALAPPDATA\Microsoft\Windows\PowerShell and restart PowerShell, does this help at all? Note: I did this and had to wait around 15 seconds for the cache file to reappear. It also took a few seconds more for the autocomplete to work.

My understanding is that you can also force a refresh of the cache with Get-Module -ListAvailable -Refresh.

Something else that springs to mind is that I noticed the module contained a number of DLL files. Could any of these be blocked on your systems? (right click > properties is a quick way to check this).

[MaxFalcone]

Hi @robinmalik, thanks for help. This didn't work, but I managed to find solution. CmdletsToExport with explicit names ('*' won't do) should be used to allow module to be imported automatically. I'll report this to original repository. Have no issue with Pode then!

[Badgerati]

Hi @MaxFalcone,

You've actually stubbled over the same bug someone on Discord found, but with the MicrosoftTeams module :)

There is actually a bug in Pode that causes this to happen; for Teams it was a case that letting PowerShell auto-import the module worked as the CmdletsToExport was set properly. While in PSLiteDB, you're right that * actually does nothing 😂

The bug is in Pode's auto-import logic, to load imported modules into its runspaces. It goes off the module's Path, which for Teams was the .psm1 and for PSLiteDB it appears to be a .dll. Since the .psd1 isn't being imported, a lot of extras are missed - such as the entire psm1 for PSLiteDB! 😳

I have a fix for it, but didn't have chance to write it up with it being holidays; but it's to change the auto-import logic to look for <module_name>.psd1 otherwise fallback.

I'll re-open this issue, since it's the same, and contains another module that highlights the problem :)

[Veers01]

I think I will follow this bug since I'm the one with trouble in microsoftteams :) The workaround of not loading the module dunamically work but I ran on another problem. I can't load the module and run it in more than 4 or 5 scriptblock. After that, MS consider that I reach the concurent connetion limit... Using diconnect-microsoftteams dosen't change anything... For now, I think I'm going to create two or three users to connect...

[Badgerati]

Hi @MaxFalcone and @Veers01,

I've pushed a fix in the above commit for importing modules, I've testing with both and for me they now both seem to be importing properly. Would either of you be able to test the changes locally and verify?

@Veers01, I had a quick look on the MicrosoftTeams connection limit, and looks like it suffers from similar Session issues to AWS (though AWS did it a little more sensibly). It seems the MT module creates a new session every time, and disconnecting then reconnecting doesn't properly close down and re-use existing connections, and you have to manually call Remove-PSSession. I've thought about seeing if it's possible to build to session re-use logic in Pode, make it simpler, but will need to looking into.

For AWS, the fix was to create a session outside of Pode, and Import-PSSession (I think it was Import) into each of Pode's runspace using State/Middleware and then check the PSSession state. If the session is closed, Remove it and Connect again - but you have to do all of this with locking, etc. so 2 runspaces don't try to create the same session/use it while closed. It's quite painful! 🙈

[robinmalik]

I can help test, if you're able to specify where we should be putting our Connect-MicrosoftTeams related code?

(As an aside, I think you've another issue open re: ExchangeOnlineManagement and sessions, which I did test and it failed ambiguously after about 5-6 hours but I had logging off so didn't collect any useful information - will repeat it soon).

[Badgerati]

In the example @Veers01 sent me, there was a Connect-MicrosoftTeams within the Start-PodeServer itself and then another Connect-MicrosoftTeams call within a route, with the Import-Module MicrosoftTeams outside of Start-PodeServer. The server level call worked, but the route one failed. (it was actually in a Pode.Web page, but a route did the same).

Ah yes! There was that issue with sessions as well. It's a tricky one with sessions because they limit you to one that you have to share across all runspaces 😕 getting the thread-safety down to a T is quite awkward, which is why I'm wondering there's anything I could write to make it much easier.

[Veers01]

I can test it but I'm not sure how to use the dev version of pod. Sorry, I'm not very used to github and built tools...

[Veers01]

@Badgerati: Ok, I manage to have it working (just replace the 3 modified files on my test computer) and it seem to work. I still have a error when I use connect-microsoftteams inside the route:

Level: Error
ThreadId: 1
Server: DESKTOP-OUPM2GU
Category: InvalidOperation: (System.Collections.…oint,System.String]:Dictionary`2) [ConvertTo-Json], InvalidOperationException
Message: The type 'System.Collections.Generic.Dictionary`2[[Microsoft.TeamsCmdlets.Powershell.Connect.Common.Endpoint, Microsoft.TeamsCmdlets.PowerShell.Connect, Version=1.1.5.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
StackTrace: at Write-PodeJsonResponse, C:\Users\BD\Documents\PowerShell\Modules\Pode\2.5.1\Public\Responses.ps1: line 717
at <ScriptBlock>, <No file>: line 11
at Invoke-PodeScriptBlock, C:\Users\BD\Documents\PowerShell\Modules\Pode\2.5.1\Public\Utilities.ps1: line 697
at <ScriptBlock>, <No file>: line 114

Date: 2022-01-02 22:28:08
Level: Error
ThreadId: 1
Server: DESKTOP-OUPM2GU
Category: System.Management.Automation
Message: The type 'System.Collections.Generic.Dictionary`2[[Microsoft.TeamsCmdlets.Powershell.Connect.Common.Endpoint, Microsoft.TeamsCmdlets.PowerShell.Connect, Version=1.1.5.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]' is not supported for serialization or deserialization of a dictionary. Keys must be strings.
StackTrace:    at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)

For the max session and the problem with deconnection, I check if I have any session open with get-pssession when I use connect-microsoftteams and I don't. I'm not sure that the new version of connect-microsoftteams (vs the skypeforbusiness module) still use a pssession or a web API in the background.

I still hope that Microsoft release more functions in the MS Graph API for Teams (There is almost noting for the voice part) because it so much easier to use and way more fast that the PSmodule.

[robinmalik]

I copied @Veers01's approach here as I was unable to get Invoke-Build to work, but with the latest updates this works for me. Full code for ease of following (note that Connect-LUMicrosoftTeams is just a custom wrapper function that ultimately calls Connect-MicrosoftTeams - we have a number of these for different MS services):

server.ps1:

Import-Module MicrosoftTeams

Start-PodeServer {

    New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging -Level Error, Debug, Verbose

    Set-PodeViewEngine -Type Pode

    Enable-PodeSessionMiddleware -Duration 1200 -Extend

    ########################################################################################
    #Region Server Level Thread Safe 'state' objects
    Lock-PodeObject -ScriptBlock {
        Set-PodeState -Name 'ApplicationRoot' -Value @{ 'ApplicationRoot' = $PSScriptRoot } | Out-Null
    }
    #EndRegion Server Level Thread Safe Objects

    Connect-LUMicrosoftTeams -Tenancy abc123 -Verbose

    ########################################################################################
    #Region Additional state objects
    #EndRegion Additional state objects

    ########################################################################################
    #Region Listeners
    Add-PodeEndpoint -Address (Get-PodeConfig).Url -Port $(Get-PodeConfig).Port -Protocol Http
    #EndRegion Listeners

    ########################################################################################
    #Region Schedules
    #EndRegion Schedules

    ########################################################################################
    #Region Routes
    Use-PodeRoutes -Path "routes"

} -Threads 5

routes/team.ps1:

Add-PodeRoute -Method Get -Path '/team/:id' -ScriptBlock {
    Import-Module MicrosoftTeams
    $Value = $WebEvent.Parameters['id']
    # Suppress 'Fetching Teams' progress message that sticks in the shell window and never vanishes
    $ExistingPP = $ProgressPreference
    $ProgressPreference = "SilentlyContinue"
    $Data = Get-Team -MailNickName $Value | Where-Object { $_.MailNickname -eq $Value }
    $ProgressPreference = $ExistingPP
    Write-PodeJsonResponse -Value $Data
}

Yesterday this didn't work, so the changes made in #892 seem to have resolved things for me 👊🏻

[robinmalik]

A few observations/questions:

1. The server is now incredibly slow to start. I've noticed MicrosoftTeams version 3.0.0 is much slower to initiate but it doesn't appear to be with the connection part. The wrapper function outputs: VERBOSE: 03/01/2022 11:16:28: Selected account: m365ad2-svc VERBOSE: 03/01/2022 11:16:28: Connecting. VERBOSE: 03/01/2022 11:16:31: Connected.

... and then it's a fair while longer until the Listening on the following 1 endpoint(s) [5 thread(s)]: appears. I don't think we can do much about this though. Using the Graph API directly where possible is much faster but for some things, the Teams module is needed.

2. Is there any way we can call Disconnect-MicrosoftTeams when the server terminates? If I terminate the server with CTRL+C and run a Get-Team, it's still connected. I haven't seen any detrimental impact at the moment when running server.ps1 again, but generally it'd be good to clean up if we can.

[Veers01]

@robinmalik: Yes,MS Teams module is verrrryyyy slow to connect, This have noting to do with pode, I'it slow in the command line too. I have two tenant and I have the same excruciating slow speed to connect.

You can call disconnect, but like @Badgerati said, this is not working as intented.

For my part, I get around by making another small pode script who act as a REST API. The script connect to teams with connect-microsoftteams and listen to a localhost socket for a url (like localhost:8080/users/ID). A each request, I do a small "get-tenant" to check if the connection is still open. On error, I reconnect teams. This is the only way I can have a reasonable response time, at the cost of more data transfomation).

Maybe there is a way to do that with a runspace or other technique, but this is not something that I'd work with so that could take some time :)

[robinmalik]

@Veers01 Multiple tenancies too and we have the same. Pretty sure it's started happening since v2.5+ but given v3 will be the only supported path going forwards there's little we can do but work with/around it.

Interesting approach re: separate Pode server for the Teams bit. There's some code in #841 regarding Exchange sessions and reconnection that might help integrate the Teams code into your "main" Pode server. So far I've not done anything to deal with Teams disconnections... nor have I hit any kind of maximum connection limits for our tenancy, but given the Connect-MicrosoftTeams is within Start-PodeServer, I assume it's just connecting once anyway.

[Badgerati]

@Veers01, I was using v3.0.0 in PS7.2.1, and when I did a Get-PSSession I was getting sessions back for MicrosoftTeams 😮 It wasn't in 5.1, which is weird, possibly 3.0.1 is different 🤔

With regards to the Write-PodeJsonResponse error, it looks like you're passing a Dictionary with non-string keys? ConvertTo-Json requires all keys in a Dictionary/Object/Hashtable to be strings :)

@robinmalik Ooh yeah, as you've both mentioned that MicrosoftTeams module is slooow. There's not much I can do about it directly, however! I've been looking into seeing if it's possible to start all runspaces async, rather than sync - so that should "speed" it up a bit. It does look like it's possible, just need to play/test with it.

And yep, there is! If you register an event for server "Terminate" or "Stop", then you should be able to run Disconnect-MicrosoftTeams and/or Remove-PSSession there 😄

[Veers01]

@Badgerati

@Veers01, I was using v3.0.0 in PS7.2.1, and when I did a Get-PSSession I was getting sessions back for MicrosoftTeams 😮 It wasn't in 5.1, which is weird, possibly 3.0.1 is different 🤔

Weird, I do have the same version:

PS C:\Users\benjamin.duval.BUREAU2> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Microsoft Windows 10.0.18363
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

PS C:\Users\benjamin.duval.BUREAU2> get-module

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Manifest   7.0.0.0               Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-Item, Clear-It…
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Ob…
Script     3.0.0                 microsoftteams                      {Add-TeamChannelUser, Add-TeamUser, Connect-Micro…
Script     2.1.0                 PSReadLine                          {Get-PSReadLineKeyHandler, Get-PSReadLineOption, …
Script     0.0                   SetMSTeamsReleaseEnvironment

PS C:\Users\benjamin.duval.BUREAU2> Get-PSSession
PS C:\Users\benjamin.duval.BUREAU2> connect-microsoftteams

Account                          Environment Tenant                               TenantId
-------                          ----------- ------                               --------
benjamin.duval@xxxxxxxxxxxxxxxx.yyy AzureCloud  xxxxxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx xxxxxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx

PS C:\Users\benjamin.duval.BUREAU2> Get-PSSession
PS C:\Users\benjamin.duval.BUREAU2>

With regards to the Write-PodeJsonResponse error, it looks like you're passing a Dictionary with non-string keys? ConvertTo-Json requires all keys in a Dictionary/Object/Hashtable to be strings :)

I dont parse anythig, this is the output of "connect-microsoftteams" :(

[Veers01]

After some more testiing, I think that the teams session is still valid in the podewebpage. I change my test code for that:

Import-Module Pode
Import-Module Pode.Web
import-module microsoftteams

Start-PodeServer {

    Use-PodeWebTemplates -Title 'Example' -Theme Dark

    Add-PodeEndpoint -Address * -Port 80 -Protocol Http

    # Logging dans la console
    New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging

    $user = "teams_svc@xxxxxxxxxxxxxxx.onmicrosoft.com"
    $pass = ConvertTo-SecureString -String 'xxxxxxxxxxxxxxxxxxxxx' -AsPlainText -Force
    $cred = New-Object System.Management.Automation.PSCredential($user,$pass)

    #### this one is working #####
    Connect-MicrosoftTeams -Credential $cred
    $test = Get-CsOnlineUser -identity testmail@xxxxxxxx.net
    write-host $test.SipAddress
    ##############################

    Add-PodeWebPage -Name "Liste" -Icon 'phone' -ScriptBlock {

        new-PodeWebContainer -Id 'searchlist' -NoBackground -Content @(
            New-PodeWebForm -Name 'Recherche' -ID 'recherchePS' -AsCard -ScriptBlock {

                $searchValue = $WebEvent.Data.Recherche

                    if($searchValue -ne $null){
            get-module | out-default
            #export-podeModule MicrosoftTeams
                    #import-module microsoftteams
                    $user = "teams_svc@xxxxxxxxxxxxxxx.onmicrosoft.com"
                    $pass = ConvertTo-SecureString -String 'xxxxxxxxxxxxxxxxxxxxx' -AsPlainText -Force
                    #$cred = New-Object System.Management.Automation.PSCredential($user,$pass)

                    # This one was not working but work with change from 892 #####
                    #Connect-MicrosoftTeams -Credential $cred | out-null
                    $test = Get-CsOnlineUser -identity estmail@xxxxxxxx.net
                    write-host $test.SipAddress
                    ###############################

                    ## code for updating the table
                }

            } -Content @(
                New-PodeWebTextbox -Name 'Recherche' -Type Text
            )

            New-PodeWebTable -Id 'listeposte' -DataColumn 'Nom' -Name 'Liste des postes' -AsCard -Paginate -PageSize 20 -ScriptBlock {

            } `
            -Columns @(
                Initialize-PodeWebTableColumn -Key "Nom" -Alignment Left
                Initialize-PodeWebTableColumn -Key "Téléphone" -Alignment Left
                Initialize-PodeWebTableColumn -Key "Extension" -Alignment Center
                Initialize-PodeWebTableColumn -Key "Carnet" -Alignment Left
                Initialize-PodeWebTableColumn -Key 'Plan de num.' -Alignment Center
                Initialize-PodeWebTableColumn -Key 'Actions' -Alignment Center
            )
        )
    }
}

And the get-csonlineuser inside the podewebpage work! I don't have to reconnect. I will make a more elaborate script with some connection check (for timeout) and get back to you.

[robinmalik]

@Veers01 Hey, just a quick reply to say I think there's a password and username in this above code about half way down and if so, you might want to delete the comment and re-reply! And change the password for good measure perhaps :) If I'm wrong, apologies.

[Veers01]

@robinmalik : You are right, I forgot this one :) Thank you for report that! At least this is just a simple account on our test tenant!

[Badgerati]

I wonder if it creates a session elsewhere, like how az/kubectl do theirs. I just tried something as well, the session isn't created in the main terminal, but when Connect-MicrosoftTeams and Get-CsOnlineUser are called in the runspace/route that does create a session 🤔

I'm going to push the main fix for the module loading issue, and close this ticket. Probably worth moving the sessions talk to Discord 😄