Closed gijs007 closed 5 years ago
Do you have the full AppService log file that you could email me?
asdkconfigurator @ outlook . com - I’m curious if there’s anything else hidden in there.
Is the environment internet connected without a proxy or anything like that?
Thanks!
I've send the log files.
The server is hosted in a datacenter and has a corporate firewall in front of it. I haven't had any issues with the normal online deployment method, so I doubt that is causing any issues.
Note that: I'm not sure if this is a structural error. It could very well be that a redeploy would work fine, if issue #59 is resolved.
In that case, there should simply be some sort of retry for the VM deployment.
OK, after reviewing your logs, I don't think your original issue was due to the database. WHile there are many points in the large log file that say "Waiting for Hosting database to be ready", on all occasions, eventually, that was successful.
The error you have is around provisioning of CN0-VM, specifically "failed to provision vm extensions for vm cn0-vm" which i think may have been a random transient issue. I'm not sure how the App Service handles reinstalling over a partially failed attempt, so what I may do is write some cleanup of the resource group itself, and if there has been a failed run previously, I just completely clean the App Service resource group (and the SQL DB cleanup) and then it tries a fresh run again.
I'll work on that later.
So my challenge is, I don't know how the AppService.exe handles the re-run. We know it fails almost immediately if the database exists, and so we have accounted for that by cleaning the DB. But, if you look in the appservice-infra resource group, and click deployments, you'll see that there are 9 separate deployments that take place:
AppService.DeployCloud SharedWorkerTierScaleSetDeploy LargeWorkerTierScaleSetDeploy SmallWorkerTierScaleSetDeploy MediumWorkerTierScaleSetDeploy AppService.DeployTenantHydrationStorage AppService.DeployAdminHydrationStorage AppService.DeployUsageStorage AppService.DeployStorage
I don't know at what stage, any of these deployments write to the DB. In a default deployment on the ASDK, 7 of the 9 deployments listed above (from LargeWorker down to DeployStorage) finish in less than 2 minutes. If all of those succeed, but say, the AppService.DeployCloud fails, if we're cleaning up the database before the next attempt, will that break anything associated with the steps that had previously completed? That's the question I don't know.
It may be 'safer' and more predictable, to completely delete the appservice-infra resource group if it already exists, and then start a fresh for each attempt. I'll look into it a bit more though.
Let's retest with the latest 1809.2 version of the DeployAppService.ps1 - as mentioned here: https://github.com/mattmcspirit/azurestack/issues/59#issuecomment-439580037 - this should clean up failed runs and retry from the start again.
Unfortunately it still fails with the new script. I've send the log to your email address.
Yep, looking at your logs, my script is not catching the fact that you have a failed deployment within the appservice-infra resource group. Can you confirm that you still see the appservice-infra RG and that there is a failed deployment within it?
I'll work on adjusting the code to make it work.
Could you run this in your environment?
# Check if there is a previous failure for the App Service deployment - easier to completely clean the RG and start fresh
# Login to Azure Stack first
$azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
$appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra" -Name "AppService.DeployCloud" -ErrorAction Stop)
$appServiceFailCheck.ProvisioningState
if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting cleanup..."
}
and post the output here? Does it print the 'There is evidence...' message?
The appservice-infra resource group doesn't exist. I do see appservice-fileshare and appservice-sql resource groups.
I just ran the commands you send, but it doesn't seem to print. The variable $appServiceFailCheck.ProvisioningState doesn't seem to be set.
PS C:\Windows\system32> # Check if there is a previous failure for the App Service deployment - easier to completely cle
an the RG and start fresh
PS C:\Windows\system32> # Login to Azure Stack first
PS C:\Windows\system32> $azsLocation = (Get-AzsLocation).Name
WARNING: Preview version of the module Azs.Subscriptions.Admin loaded. Future release of this module may have breaking
changes.
New-ServiceClient : Run Login-AzureRmAccount to login.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Generated.PowerShell.Commands\SwaggerPathCo
mmands\Get-AzsLocation.ps1:61 char:33
+ ... SubscriptionsAdminClient = New-ServiceClient @NewServiceClient_params
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : AzureRmContextError,New-ServiceClient
The property 'Locations' cannot be found on this object. Verify that the property exists.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Generated.PowerShell.Commands\SwaggerPathCo
mmands\Get-AzsLocation.ps1:66 char:9
+ $TaskResult = $SubscriptionsAdminClient.Locations.ListWithHtt ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], PropertyNotFoundException
+ FullyQualifiedErrorId : PropertyNotFoundStrict
PS C:\Windows\system32> $ArmEndpoint = "https://adminmanagement.local.azurestack.external"
PS C:\Windows\system32> Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Name : AzureStackAdmin
EnableAdfsAuthentication : False
OnPremise : False
ActiveDirectoryServiceEndpointResourceId : https://adminmanagement.XXXX.onmicrosoft.com/XXXX
AdTenant :
GalleryUrl : https://adminportal.local.azurestack.external:30015/
ManagementPortalUrl :
ServiceManagementUrl :
PublishSettingsFileUrl :
ResourceManagerUrl : https://adminmanagement.local.azurestack.external
SqlDatabaseDnsSuffix :
StorageEndpointSuffix : local.azurestack.external
ActiveDirectoryAuthority : https://login.microsoftonline.com/
GraphUrl : https://graph.windows.net/
GraphEndpointResourceId : https://graph.windows.net/
TrafficManagerDnsSuffix :
AzureKeyVaultDnsSuffix : vault.local.azurestack.external
DataLakeEndpointResourceId :
AzureDataLakeStoreFileSystemEndpointSuffix :
AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix :
AzureKeyVaultServiceEndpointResourceId : https://vault.local.azurestack.external
AzureOperationalInsightsEndpointResourceId :
AzureOperationalInsightsEndpoint :
VersionProfiles : {}
ExtendedProperties : {}
BatchEndpointResourceId :
PS C:\Windows\system32> Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
Account : XXXXX@XXXXX.onmicrosoft.com
SubscriptionName : Default Provider Subscription
SubscriptionId : XXXX
TenantId : XXXX
Environment : AzureStackAdmin
PS C:\Windows\system32>
PS C:\Windows\system32> $appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra"
-Name "AppService.DeployCloud" -ErrorAction Stop)
Get-AzureRmResourceGroupDeployment : Resource group 'appservice-infra' could not be found.
At line:1 char:25
+ ... ailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "ap ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Get-AzureRmResourceGroupDeployment], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.GetAzureResourceGroupDep
loymentCmdlet
PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState
PS C:\Windows\system32>
PS C:\Windows\system32> if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
>> Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. St
arting cleanup..."
>> }
PS C:\Windows\system32> if ($($appServiceFailCheck.ProvisioningState) -eq 'Failed') {
>> Write-Host "There is evidence of a previously failed App Service deployment in the App Service Resource Group. Starting clean
up..."}
PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState)
At line:1 char:39
+ $appServiceFailCheck.ProvisioningState)
+ ~
Unexpected token ')' in expression or statement.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : UnexpectedToken
PS C:\Windows\system32> $appServiceFailCheck.ProvisioningState
OK, this is strange:
PS C:\Windows\system32> $appServiceFailCheck = (Get-AzureRmResourceGroupDeployment -ResourceGroupName "appservice-infra"
-Name "AppService.DeployCloud" -ErrorAction Stop)
Get-AzureRmResourceGroupDeployment : Resource group 'appservice-infra' could not be found.
If you look in the admin portal, under Resource Groups, is there one that shows "appservice-######"?
What is the result when you run:
Get-AzureRmResourceGroup
I don't have a resource group which is literally called "appservice-######" As mentioned earlier, I do have the appservice-fileshare and appservice-sql resource groups.
Output from Get-AzureRmResourceGroup
:
ResourceGroupName : azurestack
Location : westcentralus
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/azurestack
ResourceGroupName : cloud-shell-storage-westeurope
Location : westeurope
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/cloud-shell-storage-westeurope
ResourceGroupName : RDGW
Location : westeurope
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/RDGW
The output above, is from Azure. You need to make sure you're logged into Azure Stack first, then run Get-AzureRmResourceGroup.
So do this:
$azsLocation = (Get-AzsLocation).Name
$ArmEndpoint = "https://adminmanagement.local.azurestack.external"
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
Get-AzureRmResourceGroup
And see what comes back.
I'm curious why you don't have the appservice-infra, and if there's another random one that's been created. Aside from the appservice-sql and appservice-fileshare (which my script creates for you), if you have no other appservice-
The output from that command is:
PS C:\Windows\system32> $azsLocation = (Get-AzsLocation).Name
WARNING: Preview version of the module Azs.Subscriptions.Admin loaded. Future release of this module may have breaking
changes.
Get-Exception : Code = InvalidResourceNamespace
Message = The resource namespace 'Microsoft.Subscriptions.Admin' is invalid.
At C:\Program Files\WindowsPowerShell\Modules\Azs.Subscriptions.Admin\0.3.0\Get-TaskResult.ps1:99 char:21
+ Get-Exception -Exception $ex
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], Exception
+ FullyQualifiedErrorId : System.Exception,Get-Exception
PS C:\Windows\system32> $ArmEndpoint = "https://adminmanagement.local.azurestack.external"
PS C:\Windows\system32> Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "$ArmEndpoint" -ErrorAction Stop
Name : AzureStackAdmin
EnableAdfsAuthentication : False
OnPremise : False
ActiveDirectoryServiceEndpointResourceId : https://adminmanagement.masdemo.onmicrosoft.com/XXXX
AdTenant :
GalleryUrl : https://adminportal.local.azurestack.external:30015/
ManagementPortalUrl :
ServiceManagementUrl :
PublishSettingsFileUrl :
ResourceManagerUrl : https://adminmanagement.local.azurestack.external
SqlDatabaseDnsSuffix :
StorageEndpointSuffix : local.azurestack.external
ActiveDirectoryAuthority : https://login.microsoftonline.com/
GraphUrl : https://graph.windows.net/
GraphEndpointResourceId : https://graph.windows.net/
TrafficManagerDnsSuffix :
AzureKeyVaultDnsSuffix : vault.local.azurestack.external
DataLakeEndpointResourceId :
AzureDataLakeStoreFileSystemEndpointSuffix :
AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix :
AzureKeyVaultServiceEndpointResourceId : https://vault.local.azurestack.external
AzureOperationalInsightsEndpointResourceId :
AzureOperationalInsightsEndpoint :
VersionProfiles : {}
ExtendedProperties : {}
BatchEndpointResourceId :
PS C:\Windows\system32> Add-AzureRmAccount -EnvironmentName "AzureStackAdmin" -ErrorAction Stop
Account : XXXX@XXXX.onmicrosoft.com
SubscriptionName : Default Provider Subscription
SubscriptionId : XXXX
TenantId : XXXX
Environment : AzureStackAdmin
PS C:\Windows\system32> Get-AzureRmResourceGroup
ResourceGroupName : appservice-fileshare
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/appservice-fileshare
ResourceGroupName : appservice-sql
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/appservice-sql
ResourceGroupName : azurestack-activation
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/azurestack-activation
ResourceGroupName : azurestack-dbhosting
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/azurestack-dbhosting
ResourceGroupName : azurestack-images
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/azurestack-images
ResourceGroupName : system.keyvault
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.keyvault
ResourceGroupName : system.local
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.local
ResourceGroupName : system.local.adminkeyvault
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.adminkeyvault
ResourceGroupName : system.local.AzureMonitor
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.AzureMonitor
ResourceGroupName : system.local.dbadapter.dns
Location : local
ProvisioningState : Succeeded
Tags :
Name Value
============ ====================
ProviderName SQLAdapter
Version 1.1.24.0
Category Foundation
Role DNSZoneResourceGroup
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.dbadapter.dns
ResourceGroupName : system.local.keyvault
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.keyvault
ResourceGroupName : system.local.MDM
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.MDM
ResourceGroupName : system.local.sqladapter
Location : local
ProvisioningState : Succeeded
Tags :
Name Value
============ =============
NextVersion
Version 1.1.24.0
Category Foundation
Role ResourceGroup
LastVersion
ProviderName SQLAdapter
ResourceId : /subscriptions/XXXX/resourceGroups/system.local.sqladapter
ResourceGroupName : System.UsageConnection
Location : local
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/XXXX/resourceGroups/System.UsageConnection
OK, so that shows you have no left-over appservice-infra stuff, so there shouldn't be a problem re-deploying the App Service.
As a test, i crashed the App Service deployment within my new ASDK. Within the appservice-infra resource group, mine clearly showed 'failed' for the AppService.DeployCloud deployment. I then reran the script. My script successfully cleaned up the resuorce group, and successfully redeployed the App Service, so it must be something in your environment.
Can you try one more time, ensuring you've downloaded this version: https://raw.githubusercontent.com/mattmcspirit/azurestack/1809.2/deployment/powershell/DeployAppService.ps1 and copied it into your ConfigASDK\Scripts folder, overwriting the one that's there.
Then run the main script and see if it fails.
NOTE on your launch script, enclose your passwords in single quotes 'P@ssw0rd!' rather than double-quotes.
If it does fail, can you please email me the log file from the app service itself (not the log files in the ConfigASDK\Logs folder). This appservice log file will be within the downloadPath\ASDK\appservice folder. The name of the log will be something like AppServiceLog1118-145337 but your numbers will be different.
If it still fails with the 'network password' issue, we'll do some additional cleanup and try again.
I've send you the logs, as it still fails with the 'network password' issue. I've tried enclosing the passwords in single and double-quotes, but this makes no difference.
OK, here's what we're going to do.
So, we've essentially cleaned up your temp folder, cleaned up certs and old txt files, and removed your app. We need to ensure that the script will run the 'AppServicePreReqs stage again, so we'll need to do this:
$sqlServerInstance = '(localdb)\MSSQLLocalDB'
$databaseName = "ConfigASDK"
$tableName = "Progress"
$progressStage = "AddAppServicePreReqs"
Invoke-Sqlcmd -Server $sqlServerInstance -Query "USE $databaseName UPDATE Progress SET $progressStage = 'Incomplete';" -Verbose:$false -ErrorAction Stop
Read-SqlTableData -ServerInstance $sqlServerInstance -DatabaseName "$databaseName" -SchemaName "dbo" -TableName "$tableName" -ErrorAction SilentlyContinue -Verbose:$false
Then run it in the ISE. The output from the last command should show the AddAppServicePreReqs stage as 'Incomplete'.
Did that work? If so, close the ISE, ensure all PS windows are closed, then open a fresh PS Console/ISE and you should be good to try a rerun of the main script, and remember to use single quotes for containing passwords.
I followed all those steps, however the rerun didn't go very well:
Shortly after it stopped running. Seems like it's not finding the script files, even thought they are there:
****** Please wait until all jobs have completed/failed before re-running the main script ******
At least one of the jobs failed.
FAILED JOB: Job Name: AddServerCoreImage | Error Message: The term '.\Scripts\AddImage.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddServerFullImage | Error Message: The term '.\Scripts\AddImage.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLAzpkg | Error Message: The term '.\Scripts\AddGalleryItems.ps1' is not recognized as the n
ame of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerAzpkg | Error Message: The term '.\Scripts\AddGalleryItems.ps1' is not recognized as t
he name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was includ
ed, verify that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLRP | Error Message: The term '.\Scripts\DeployDBRP.ps1' is not recognized as the name of a
cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify t
hat the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerRP | Error Message: The term '.\Scripts\DeployDBRP.ps1' is not recognized as the name
of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, veri
fy that the path is correct and try again.
FAILED JOB: Job Name: AddMySQLSku | Error Message: The term '.\Scripts\AddDBSkuQuota.ps1' is not recognized as the name
of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, veri
fy that the path is correct and try again.
FAILED JOB: Job Name: AddSQLServerSku | Error Message: The term '.\Scripts\AddDBSkuQuota.ps1' is not recognized as the n
ame of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included,
verify that the path is correct and try again.
FAILED JOB: Job Name: UploadScripts | Error Message: The term '.\Scripts\UploadScripts.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: DeployMySQLHost | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the name o
f a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verif
y that the path is correct and try again.
FAILED JOB: Job Name: DeploySQLServerHost | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the na
me of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, v
erify that the path is correct and try again.
FAILED JOB: Job Name: DeployAppServiceFS | Error Message: The term '.\Scripts\DeployVM.ps1' is not recognized as the nam
e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, ve
rify that the path is correct and try again.
FAILED JOB: Job Name: AddAppServicePreReqs | Error Message: The term '.\Scripts\AddAppServicePreReqs.ps1' is not recogni
zed as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path wa
s included, verify that the path is correct and try again.
FAILED JOB: Job Name: DeployAppService | Error Message: The term '.\Scripts\DeployAppService.ps1' is not recognized as t
he name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was includ
ed, verify that the path is correct and try again.
Please review the logs for further troubleshooting
At C:\ConfigASDK\Scripts\GetJobStatus.ps1:94 char:5
+ throw "Please review the logs for further troubleshooting"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (Please review t...troubleshooting:String) [], RuntimeException
+ FullyQualifiedErrorId : Please review the logs for further troubleshooting
PS C:\ConfigASDK>
This is bizarre, and I have no idea why that wouldn't work. There is no reason for that to fail.
Yes, that is all the case.
I'm retrying it again in a new PowerShell window again. Instead of using the CD command I used set-environment, I'm not sure if that matters.
Seems to work sort of better:
It failed at the VM provisioning again, just like it did when I opened this issue.
Could you email the AppService log from within the ASDK\appservice folder? I don’t think think this is script related. Either something is wrong with the base Windows Server 2016 Datacenter image, or the virtualization stack. I’ll review the logs and if required, share with the App Service team.
I've send the log file.
OK, I've sent it to out internal App Service on Azure Stack team, to help us figure out what's happening here. I'll follow up when I have more info, but don't delete anything just yet, so we can investigate further. Thanks!
I think we can close this issue here? Since it also happens when installing App Services manually, so it's not script related.
Then again, you may want to keep it open to avoid other people from reporting the same issue ;)
Closed for now, we can always reopen later :)
I've ran the post deployment script on a fresh ASDK with the following arguments:
The post deployment step for DeployAppService failed. After checking the AppServiceLog it, it seems that a virtual machine is being deployed. But the deployment of this virtual machine failed.
From the: AppServiceLog1116-011218.txt: