Populate local workspace - weird behaviour

[romankua]

Hello! I'm looking forward trying SPGO, but struggling with configuring it.

I work with SP2013 on-premises using NTLM (Kerberos) authentication. Here is my spgo.json

{
    "sourceDirectory": "src",
    "sharePointSiteUrl": "https://shpdev",
    "publishingScope": "None",
    "authenticationType": "NTLM",
    "remoteFolders": [
        "/Style Library/portal/**/*.*"
    ]
}

When I issue SPGo: Populate local workspace command the following happens: SPGo console prints: Starting File Synchronization... Progress bar shows 'Populating workspace' but it doesn't end. And local src folder is not populated. Also in ULS I found that FileNotFound exception occured, here is some logs:

Name=Request (POST:https://shpdev:443/_api/Web/GetList(@DocLibUrl)/GetItems?$select=Name,UniqueID,ID,FileDirRef,FileRef,FSObjType,TimeCreated,TimeLastModified,Length,ModifiedBy&@DocLibUrl=%27%252FStyle%2520Library%252Fportal%252F**%252F%27) d23f699e-458e-60ef-2126-afb74ab7bb92 Unknown SPRequest error occurred. More information: 0x80070002 d23f699e-458e-60ef-2126-afb74ab7bb92 SPRequest.GetMetadataForUrl: UserPrincipalName=..., AppPrincipalName= ,bstrUrl=/Style Library/portal/** ,METADATAFLAGS=59 d23f699e-458e-60ef-2126-afb74ab7bb92 System.IO.FileNotFoundException: 0x80070002, StackTrace: at Microsoft.SharePoint.SPWeb.GetList(String strUrl) at... d23f699e-458e-60ef-2126-afb74ab7bb92

If I set "remoteFolders": [ "/Style Library/portal/" ] I get in SPGo console: Starting File Synchronization... Successfully downloaded 0 files to: src/Style Library/portal/ file synchronization complete. But src folder is empty.

In ULS I found error:

SPSecurityContext: Could not retrieve a valid windows identity for username 'DOMAIN\USER' with UPN 'user@domain'. UPN is required when Kerberos constrained delegation is used. Exception: System.ComponentModel.Win32Exception (0x80004005): Access is denied

But if I set "remoteFolders": [ "/Style Library/portal/*.*" ] I get: Starting File Synchronization... Successfully downloaded 175 files to: src/Style Library/portal/. file synchronization complete.

And src folder is populated with Style Library folder and everything what is inside, not only "/Style Library/portal/" folder!

So, my question is: am I doing something wrong or is this a bug?

Thank you!

[sg-chrishasz]

Hello @akupaka !

Thank you for the detailed issue report. I tested locally and can confirm there is a bug with Glob parsing when you request ".". I'll work on a fix for that.

As a temporary workaround, can you replace the instances of *.* with just a single *? That should achieve the same results.

It is also worth mentioning that setting the remoteFolders property to [ "/Style Library/portal/" ] is behaving as intended. That configuration should only download files specifically in that folder. No subdirectories or their contained files.

I'll make sure a fix goes out in the next build. Thanks for using SPGo!

[romankua]

Hello @sg-chrishasz , thx for your time! Is there any workaround to download subdirectories in specified folder?

[sg-chrishasz]

Not at the moment, but I will resolve this in 1.3.0, which I should be able to release this week.

[sg-chrishasz]

I have just published 1.3.0 to the VS Marketplace. Please confirm that the issue is resolved and we can close this issue.

Thanks!

[romankua]

Hi, Chris! So, I have some results =)

"remoteFolders": ["/Style Library/portal/"]
"remoteFolders": ["/Style Library/portal/*"]
"remoteFolders": ["/Style Library/portal/*.*"]

Any of the above downloads only files inside /Style Library/portal/ directory.

And "remoteFolders": ["/Style Library/portal/**/*.*"] downloads everything from /Style Library/ =/ Not exactly what we were expecting =)

[romankua]

@sg-chrishasz , look what I've just found! =)

Although I have no idea how VS Code extensions work, but I found SPGo ext folder and looked through files, and changed some code inside... and it worked! =)

So, I found that function spFileGateway.js/SPFileGateway.downloadFiles calls sppull passing it fileOptions variable. I printed it to console and found out that it has camlCondition property: "camlCondition":"<Contains><FieldRef Name=\"FileLeafRef\"/><Value Type=\"Text\">.</Value></Contains>" Than I changed it a bit using a hammer, but anyway =) fileOptions.camlCondition = "<Contains><FieldRef Name=\"FileLeafRef\"/><Value Type=\"Text\">portal</Value></Contains>"; And I got some files inside /Style Library/ which contain 'portal' in their path. So, I suggest looking into code which builds fileOptions.camlCondition, it must be GlobToCamlConverter class, but I'm not sure what changes should be like.

I hope this will help! =)

[romankua]

I spent some time analyzing the issue and found better CAML for my particular example. If I set fileOptions.camlCondition in downloadFiles this way: fileOptions.camlCondition = "<BeginsWith><FieldRef Name=\"FileDirRef\"/><Value Type=\"Lookup\">" + remoteFolder + "</Value></BeginsWith>"; it gets all inside remoteFolder except empty folders, which is pretty cool. Please note that CAML Contains will not work with Lookup, and CAML Eq will return only files inside given directory itself. So I use CAML BeginsWith, though I'm not sure whether this have issues with other field types.

upd1: It seems that is CAML Contains which doesn't work with FileDirRef. fileOptions.camlCondition = "<FieldRef Name=\"FileDirRef\"/><Value Type=\"Text\">" + remoteFolder + ""; gives the same result =)

upd2: I was wrong, CAML Contains works with FileDirRef in general. But it doesn't when full path begins with forward slash. So this one will work: <Contains><FieldRef Name=\"FileDirRef\"/><Value Type=\"Text\">Style Library/portal</Value></Contains> While this will not: <Contains><FieldRef Name=\"FileDirRef\"/><Value Type=\"Text\">/Style Library/portal</Value></Contains> This one will work as well, but will return some extra (similar) <Contains><FieldRef Name=\"FileDirRef\"/><Value Type=\"Text\">/portal</Value></Contains>

Upd 3 What is interesting, escaping space character <Value Type=\"Text\">/Style%20Library/portal</Value> does not work in any case. I think it is quite important to mention in readme.

[sg-chrishasz]

@akupaka thanks for helping debug this! The issue you are seeing when you set property "remoteFolders": ["/Style Library/portal/**/*.*"], SPGo will download everything from /Style Library/

The root cause of this is that I am improperly handling the globstar (/**/*.*) use case when the globstar is appended to the end of a multi-level directory tree.

I will resolve this issue for 1.3.1.

[sg-chrishasz]

I have just published SPGo 1.3.1 to the VSCode marketplace, which should finally kill all of these glob <=> caml issues. @akupaka can you confirm this has been resolved and I will close this issue.

Thanks!

[romankua]

@sg-chrishasz , unfortunatelly it doesn't work.

As I stated earlier CAML Contains for some reason doesn't work when relative path to directory starts with forward slash. So this CAML won't work:

<And><Contains><FieldRef Name="FileDirRef"/><Value Type="Text">/Style Library/portal</Value></Contains><Contains><FieldRef Name="FileLeafRef"/><Value Type="Text">.</Value></Contains></And>

But it will work if we change it to:

<And><Contains><FieldRef Name="FileDirRef"/><Value Type="Text">Style Library/portal</Value></Contains><Contains><FieldRef Name="FileLeafRef"/><Value Type="Text">.</Value></Contains></And>

I updated GlobToCamlConverter/Convert line 39 to:

camlQuery = this.combineAnd(this.queryLike(this.getFileDirRef, (spSitePath + glob.base).replace(/^\//, '')), camlQuery);

And it started working, although I'm not sure whether this approach is suitable in general.

[sg-chrishasz]

Hello @akupaka! Thanks for bearing with me on this - I want to make sure this is fully solved. Here are the test cases and results that I see. If they are different than what you see, can you give me an example of the command you are running (e.g. SPGo> Populate Workspace vs SPGo> Retrieve Folder) and a copy of your spgo.json file.

My SharePoint Folder Structure: (forgive the lame file names)

SiteAssets
|---> folder space
      |--->subfolder
           |---> content.css
           |---> file.min.css
           |---> master.css
     |---> content.css
     |---> file.css
     |---> master.css
|---> scripts
     |---> javascript.js
|---> file.css
|---> main.css
|---> test.css
|---> test.js
|---> test.min.css
|---> test.min.js
|---> test2.css

Your examples above, and my results: note: I tried all of these both with a leading slash and without a leading slash, e.g. siteassets/folder space and /siteassets/folder space "remoteFolders": ["/Style Library/folder space/"] Results: Correct!

     |---> SiteAssets
         |---> folder space
             |---> content.css
             |---> file.css
             |---> master.css

"remoteFolders": ["/siteassets/folder space/*"] Results: Correct!

     |---> SiteAssets
         |---> folder space
             |---> content.css
             |---> file.css
             |---> master.css

"remoteFolders": ["/siteassets/folder space/*.*"] Results: Correct!

     |---> SiteAssets
         |---> folder space
             |---> content.css
             |---> file.css
             |---> master.css

"remoteFolders": ["siteassets/folder space/**/*.*"] Results: Correct!

     |---> SiteAssets
         |---> folder space
             |--->subfolder
             |---> content.css
             |---> file.min.css
             |---> master.css
         |---> content.css
         |---> file.css
         |---> master.css

I've also tested a number of combinations of different file endings: "remoteFolders": ["siteassets/**/*.js"] Results: Correct!

     |---> SiteAssets
        |---> scripts
           |---> javascript.js
        |---> test.js
        |---> test.min.js

A final thing to look for - which branch are you working from? In 1.3.1 (currently merged into master), GlobToCamlConverter.ts line 39 is just a close brace. Here is what you should see for lines 36 to 42:

//check for folder scope
        if(!glob.is.globstar && camlQuery != ''){
            camlQuery = this.combineAnd(this.queryEqual(this.getFileDirRef, spSitePath + glob.base), camlQuery );
        }
        else if(glob.is.globstar && camlQuery != ''){
            camlQuery = this.combineAnd(this.queryLike(this.getFileDirRef, spSitePath + glob.base), camlQuery );
        }
        }

Thanks for helping confirm these issues and for your support of SPGo!

-Chris

[romankua]

I'm sorry Chris, I gave you wrong line number, that is because I added few lines into that file on the beginning to have console output for debugging =/

Here is how I changed that part of the file:

if (!glob.is.globstar && camlQuery != '') {
            camlQuery = this.combineAnd(this.queryEqual(this.getFileDirRef, spSitePath + glob.base), camlQuery);
        }
        else if (glob.is.globstar && camlQuery != '') {
            camlQuery = this.combineAnd(this.queryLike(this.getFileDirRef, (spSitePath + glob.base).replace(/^\//, '')), camlQuery);
        }

My spgo.json file is the same as in the my first post in this topic. I use "Populate local workspace" command.

I will try all the tests again and inform you as soon as I can =)

[romankua]

So, I deleted extension folder from my profile and installed it again to be sure all my modifications are gone. I tried to download files from Style Library and SiteAssets libraries witout any luck. Then I put my files on different site and it worked! =) I tried to use another server - it worked as well.

So, for now I believe there is some issue with that site itself, although I have no idea what could be the reason =/

I guess the issue is resolved. Thank you, Chris! =)

[romankua]

Chris, what site did you use for your tests? It seems that the problem not within particular site, but with root site =) I checked on different server and the issue is present there.

[romankua]

Look, I used powershell to look through items properties

$site = Get-SPWeb "https://shp-dev";
$styles = $site.Lists.TryGetList("Style Library");
$styles.Items[0].xml

$site2 = Get-SPWeb "https://shp-dev/sites/team";
$styles2 = $site2.Lists.TryGetList("Style Library");
$styles2.Items[0].xml

In results I can see that 1 item: ows_FileRef='2;#Style Library/XSL Style Sheets/ContentQueryMain.xsl' ows_FileDirRef='2;#Style Library/XSL Style Sheets'

2 item: ows_FileRef='2;#sites/team/Style Library/Media Player/AlternateMediaPlayer.xaml' ows_FileDirRef='2;#sites/team/Style Library/Media Player'

So, I think that Contains "/Style Library" may not find 1 item because it doesn't have leading slash. On the other hand second item have "sites/team/Style Library" so Contains "/Style Library" will find it.

How do you think, maybe that is why I'm having this issue with root sites?

[gunner13]

I got the same issue, this doesn't work "/siteassets/*/.js", everything with "**" doesn't work

[joaofxp]

Hello, i was trying to list "Style Library/*/." but this doesn't work :/ , tried with / and only but the same happens, same = nothing.

Can you help me? Thanks !

[sg-chrishasz]

@gunner13 - a proper Glob notation should be: /siteassets/*.js. This would download anything directly in the siteassets folder, with a file extension of .js. If you wanted everything, recursively, then you should use the glob notation: /siteassets/**/*.js. Can you confirm if this solves the problem?

@joaofxp - It sounds like you want to recursively download everything (files and folders) inside the Style Library folder. The correct glob notation for this is: /Style Library/**/*. In this case, the /**/ token tells the glob processor to look in all folders, and the final * token tells the glob processor to find all files.

Thanks!

[joaofxp]

@sg-chrishasz Hello, I did as you say and now it keeps "downloading files" , never finish the download, what can I do now? I left the computer downloading the whole day and nothing changed

[sg-chrishasz]

@joaofxp - Can you post your spgo.json file here? (you can blank out your tenant name for security reasons). Also, can you past any errors from the SPGO output window (CTRL+SHIFT+U)?

I'm tracking a possibly related issue with a dependent package that is not rejecting promises when an internal error occurs that may be the cause of the endlessly spinning "downloading files" menu and all of this will help me track this down. Thanks!

[joaofxp]

@sg-chrishasz Sorry I can't, my company didn't allow me

[joaofxp]

It does not show any error, just keeps "downloading" and never stops

[sg-chrishasz]

@joaofxp no problem, I understand.

without sharing any details, can you tell me how many items you have in your remoteFolders list?

[joaofxp]

remoteFolders only for master layout, others i'm using the "retrieve folder" command

[sg-chrishasz]

@joaofxp I refactored a bunch of the code which retrieves remote files in the latest SPGo release (1.3.3). Are you still seeing the same errors?

Also, can you check the "output" tab for any messages from SPGo (you can find SPGo in the drop-down on the right side of the window). Any error/debug messages from the app will display there.

[joaofxp]

@sg-chrishasz Hey, now it showed this on the console: "Error: read ECONNRESET"

[sg-chrishasz]

hello @joaofxp -

sorry for missing this last update. We are actually tracking this issue here https://github.com/readysitego/spgo/issues/68

[chrishasz]

Everyone-

SPGo 1.4.0 contained a major revamp of the Get Folder and Populate Local Workspace commands.

If anyone is still having problems, please let me know, otherwise I’ll close this issue after a week.

[chrishasz]

Closing this issue due to age.

[kalaram20]

I am using NTLM for on-prem sp 2016, populate the local workspace continuously running. No error and nothing happening. The notification shows multiple instances of "Populating workspaces". Any thoughts on this?

[kurtisdunn]

I am using NTLM for on-prem sp 2016, populate the local workspace continuously running. No error and nothing happening. The notification shows multiple instances of "Populating workspaces". Any thoughts on this?

Was this resolved? I am having the same issue.