search

gocd / gocd

GoCD - Continuous Delivery server main repository
https://www.gocd.org
Apache License 2.0
7.05k stars 973 forks source link

TFS checkout fails when material points to a single file #5427

Closed jsmythsci closed 5 years ago

jsmythsci commented 5 years ago
Issue Type
Summary

TFS checkout fails when material points to a single file

Environment

GoCD is running on a Linux server and the affected agents are running Windows Server 2012R2.

Basic environment details
Additional Environment Details

This issue has been encountered on a copy of our production system that we created to test upgrading to the latest version of GoCD. The affected pipelines all work as intended on the following system:

Agents in the working environment are also running Windows Server 2012R2 but would be using an older version of Java since they are running an older version of the agent.

Steps to Reproduce
  1. Create a new pipeline.
  2. Add a TFS material.
  3. Set the Project Path to a single file (e.g. $/TeamProjectName/AssemblyInfo.cs)
  4. Set the Destination Directory to a single file (e.g. tfs/AssemblyInfo.cs)
  5. Trigger the pipeline
Expected Results

Pipeline should check the file out of TFS and launch its first task.

Actual Results

Pipeline fails with an error like the following:

Failed while checking out into Working Folder: pipelines\MyPipeline\tfs\AssemblyInfo.cs, Project Path: $/TeamProjectName/AssemblyInfo.cs, Workspace: bb18a58c6abfa36db84643de09258c86feaa48bc711174334eaea61594b9623f, Username: BuildUser, Domain: mydomain, Root Cause: java.io.FileNotFoundException: E:\GoCD\Go Agent\pipelines\MyPipeline\tfs\AssemblyInfo.cs\$tf\5\377de3f5-a6f8-41ed-970a-9b0bbf2c1a50.gz (The system cannot find the path specified)

Possible Fix

This appears to be an extension of #183 (TFS does not create destination directory if project path is a file). It seems that TFS is trying to create a local workspace and is failing because:

  1. It creates a file object instead of a directory.
  2. It tries to create the local $tf directory inside the file object.

If TFS were to create a server workspace instead of a local one (as it does in 15.2), this issue would likely not exist.

Log snippets

Here is the stack trace from go-agent.log when the exception is thrown:

2018-11-20 17:09:00,480 ERROR [scheduler-2] DefaultGoPublisher:140 - Failed while checking out into Working Folder: pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs, Project Path: $/MyTeamProject/AssemblyInfo.cs, Workspace: 25cc73c9040762d24de5895df1c97c2ead948fa4a0c852b0fcd9efad4fa6487c, Username: myServiceAccount, Domain: mydomain, Root Cause: java.io.FileNotFoundException: E:\GoCD\Go Agent\pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs\$tf\9\6a260479-6aa8-42e4-9d56-a0a03a82cc52.gz (The system cannot find the path specified)
java.lang.RuntimeException: Failed while checking out into Working Folder: pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs, Project Path: $/MyTeamProject/AssemblyInfo.cs, Workspace: 25cc73c9040762d24de5895df1c97c2ead948fa4a0c852b0fcd9efad4fa6487c, Username: myServiceAccount, Domain: mydomain, Root Cause: java.io.FileNotFoundException: E:\GoCD\Go Agent\pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs\$tf\9\6a260479-6aa8-42e4-9d56-a0a03a82cc52.gz (The system cannot find the path specified)
    at com.thoughtworks.go.domain.materials.tfs.AbstractTfsCommand.checkout(AbstractTfsCommand.java:96)
    at com.thoughtworks.go.tfssdk.TfsSDKCommandTCLAdapter.checkout(TfsSDKCommandTCLAdapter.java:43)
    at com.thoughtworks.go.config.materials.tfs.TfsMaterial.updateTo(TfsMaterial.java:165)
    at com.thoughtworks.go.domain.MaterialRevision.updateTo(MaterialRevision.java:146)
    at com.thoughtworks.go.domain.materials.AbstractMaterialAgent.prepare(AbstractMaterialAgent.java:43)
    at com.thoughtworks.go.remote.work.BuildWork.prepareJob(BuildWork.java:174)
    at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:126)
    at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:82)
    at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:54)
    at com.thoughtworks.go.agent.AgentHTTPClientController.retrieveWork(AgentHTTPClientController.java:134)
    at com.thoughtworks.go.agent.AgentHTTPClientController.work(AgentHTTPClientController.java:109)
    at com.thoughtworks.go.agent.AgentController.loop(AgentController.java:85)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.tfs.core.clients.versioncontrol.exceptions.VersionControlException: java.io.FileNotFoundException: E:\GoCD\Go Agent\pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs\$tf\9\6a260479-6aa8-42e4-9d56-a0a03a82cc52.gz (The system cannot find the path specified)
    at com.microsoft.tfs.core.clients.versioncontrol.internal.localworkspace.LocalWorkspaceProperties.moveBaselineFolderStructure(LocalWorkspaceProperties.java:721)
    at com.microsoft.tfs.core.clients.versioncontrol.internal.localworkspace.LocalWorkspaceProperties.doBaselineFolderMaintenance(LocalWorkspaceProperties.java:1131)
    at com.microsoft.tfs.core.clients.versioncontrol.UpdateLocalVersionQueue$4.invoke(UpdateLocalVersionQueue.java:711)
    at com.microsoft.tfs.core.clients.versioncontrol.internal.localworkspace.LocalWorkspaceTransaction.execute(LocalWorkspaceTransaction.java:350)
    at com.microsoft.tfs.core.clients.versioncontrol.UpdateLocalVersionQueue.sendToServer(UpdateLocalVersionQueue.java:708)
    at com.microsoft.tfs.core.clients.versioncontrol.UpdateLocalVersionQueue.flush(UpdateLocalVersionQueue.java:461)
    at com.microsoft.tfs.core.clients.versioncontrol.UpdateLocalVersionQueue.close(UpdateLocalVersionQueue.java:409)
    at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processOperations(GetEngine.java:1054)
    at com.microsoft.tfs.core.clients.versioncontrol.engines.internal.GetEngine.processGetOperations(GetEngine.java:808)
    at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2463)
    at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2345)
    at com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace.get(Workspace.java:2335)
    at com.thoughtworks.go.tfssdk14.wrapper.GoTfsWorkspace.get(GoTfsWorkspace.java:42)
    at com.thoughtworks.go.tfssdk14.TfsSDKCommand.retrieveFiles(TfsSDKCommand.java:94)
    at com.thoughtworks.go.domain.materials.tfs.AbstractTfsCommand.checkout(AbstractTfsCommand.java:90)
    ... 23 common frames omitted
Caused by: java.io.FileNotFoundException: E:\GoCD\Go Agent\pipelines\Test-FileCheckout\tfs\AssemblyInfo.cs\$tf\9\6a260479-6aa8-42e4-9d56-a0a03a82cc52.gz (The system cannot find the path specified)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:101)
    at com.microsoft.tfs.jni.helpers.FileCopyHelper.copy(FileCopyHelper.java:151)
    at com.microsoft.tfs.core.clients.versioncontrol.internal.localworkspace.LocalWorkspaceProperties.rename(LocalWorkspaceProperties.java:1219)
    at com.microsoft.tfs.core.clients.versioncontrol.internal.localworkspace.LocalWorkspaceProperties.moveBaselineFolderStructure(LocalWorkspaceProperties.java:719)
    ... 37 common frames omitted
Code snippets/Screenshots

Here is the full XML of the pipeline used to test this issue:

    <pipeline name="Test-FileCheckout">

      <materials>

        <tfs url="http://tfs.mydomain.com:8080/tfs/DefaultCollection" username="myServiceAccount" domain="mydomain" encryptedPassword="REDACTED" projectPath="$/MyTeamProject/AssemblyInfo.cs" dest="tfs/AssemblyInfo.cs" materialName="AssemblyInfo" />

      </materials>

      <stage name="defaultStage" cleanWorkingDir="true">

        <jobs>

          <job name="defaultJob" />

        </jobs>

      </stage>

    </pipeline>
Any other info
jsmythsci commented 5 years ago

I have done some testing with different versions and it seems that this issue was introduced in version 16.12, presumably related to the TFS SDK upgrade that it included (#234).

I updated from 15.2 directly to 16.11 and confirmed that all pipelines continued to work as intended. I then updated from 16.11 to 16.12 and found that the initial test run of the test pipeline worked as intended but all subsequent runs failed.

I suppose this means that this issue might not affect elastic agents since it doesn't seem to occur the first time a pipeline is run on an agent.

I have not explicitly tested any versions between 15.2 and 16.11 or between 16.12 and 18.10 but I don't believe those tests would be valuable at this point.

I still believe that we could work around the issue by having the agent create a server workspace instead of a local one but I have yet to find any documentation on how to do that.

arvindsv commented 5 years ago

@jsmythsci Thank you for doing all that investigation!

About having the agent create a server workspace, I don't know for sure. Maybe something changed in the SDK between those versions?

Most of the code related to this seems to be around: this and this.

You might learn something by turning on debug logs. In versions such as 16.12, there was probably a log4j.properties in the agent's config/ directory.

In my view, this looks like an unsupported case. I would assume that the working directory (called a "directory") is not a file. I wouldn't be surprised if GoCD code or the TFS SDK code has a call to mkdir somewhere.

jsmythsci commented 5 years ago

@arvindsv Thank you for taking the time to respond.

Could you clarify which case you believe is unsupported? I can see 2 different examples of unusual behaviour here and I am not sure which you are referring to.

  1. TFS material's Project Path points to a single file in the TFS repository.
  2. TFS material's Destination Directory points to a single file.

I can understand the latter not being supported but, in my opinion, it is not an actual use case but rather a workaround for the bug that is documented in #183.

jsmythsci commented 5 years ago

I dug through some of the code you linked and, if I am reading things right, I believe that GoCD's code does not currently provide a mechanism to specify the workspace location (server vs local).

I think the workspace is created here, which seems to be calling this method from the SDK.

Assuming I have correctly understood how all of these pieces fit together, GoCD is hard-coded to pass null to createWorkspace()'s WorkspaceLocation parameter, causing the workspace to be created using the server's default location.

If that is true then we may be able to work around this issue by changing our TFS server's default workspace location. I will have to do more investigation to figure out what the implications of that are.

arvindsv commented 5 years ago

@jsmythsci My expectation would be that the project path points to a directory and the destination is a directory too. However, I can see that you're trying to work around #183.

What you said about the WorkspaceLocation parameter seems correct. It does seem to be passing null which makes it use the server's default.

If you have a patch for this (or at least a reasonable idea to try), which allows us to get past this, I'm happy to work with you on it to see if we can fix the issue. What would be important, in that case, is that the rest of the tests and cases work too.

I wonder how we would know, without telling it, that the project path points to a file. Not sure how it was working before.

jsmythsci commented 5 years ago

I think the simplest thing to do would be to replace the hard-coded null with a hard-coded value for creating a server workspace. The following should do it (not sure why GitHub isn't letting me upload it as an attachment):

--- GoTfsVersionControlClient.java  2018-12-04 09:07:24.432449300 -0500
+++ "GoTfsVersionControlClient - Copy.java" 2018-12-04 09:13:19.856552100 -0500
@@ -20,6 +20,7 @@
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Changeset;
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.RecursionType;
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace;
+import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.WorkspaceLocation;
 import com.microsoft.tfs.core.clients.versioncontrol.specs.version.ChangesetVersionSpec;
 import com.microsoft.tfs.core.clients.versioncontrol.specs.version.LatestVersionSpec;

@@ -57,7 +58,7 @@
     }

     public GoTfsWorkspace createWorkspace(String workspace) {
-        return new GoTfsWorkspace(client.createWorkspace(null, workspace, null, null, null, null));
+        return new GoTfsWorkspace(client.createWorkspace(null, workspace, null, WorkspaceLocation.SERVER, null, null));
     }

     public Changeset[] queryHistory(String projectPath, ChangesetVersionSpec uptoRevision, int revsToLoad) {

I'm assuming the only use-case for this code is the following workflow on GoCD agents:

  1. Create workspace.
  2. Get source at appropriate version.
  3. Destroy workspace.
  4. Move to next task.

According to Microsoft's documentation, the reasons for using a local workspace all relate to making it easier to work offline; if that never comes up in our workflow then I don't see any downside to hard-coding a server-side workspace.

I am not a developer by trade, though, and there may very well be things I have not thought of.

I am interested to know what other people think.

jsmythsci commented 5 years ago

@arvindsv Any feedback on the patch or the thought process around it? Are there any use-cases other than fetching the file(s) from source at pipeline run time that need to be considered?

Unfortunately I don't know how to set up a development environment so I can't apply the patch and run the tests myself at this point. I could take the time to figure it out but it would be nice to know whether or not I am on the right track before making that commitment.

arvindsv commented 5 years ago

@jsmythsci I'll apply the patch and make an installer available to you tomorrow (or later today). Maybe you can set it up and take a look?

If that works for you, I'll look at what it can affect. My guess is that if multiple agents (maybe in the same machine) are all trying to use the same server workspace, it could be a problem. We will figure out what to do, in that case. For a start, let's go with the patch you have and see if it helps in any way.

arvindsv commented 5 years ago

@jsmythsci Here are the links:

Server

Agent

You will need both. You can't use this agent with your current server. Bring up the server first and then the agent and once they're connected to each other, you can try this out.

I had to change the patch a bit:

diff --git a/tfs-impl/tfs-impl-14/src/main/java/com/thoughtworks/go/tfssdk14/wrapper/GoTfsVersionControlClient.java b/tfs-impl/tfs-impl-14/src/main/java/com/thoughtworks/go/tfssdk14/wrapper/GoTfsVersionControlClient.java
index 81fdadbe9..f04b63249 100644
--- a/tfs-impl/tfs-impl-14/src/main/java/com/thoughtworks/go/tfssdk14/wrapper/GoTfsVersionControlClient.java
+++ b/tfs-impl/tfs-impl-14/src/main/java/com/thoughtworks/go/tfssdk14/wrapper/GoTfsVersionControlClient.java
@@ -20,6 +20,7 @@ import com.microsoft.tfs.core.clients.versioncontrol.VersionControlClient;
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Changeset;
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.RecursionType;
 import com.microsoft.tfs.core.clients.versioncontrol.soapextensions.Workspace;
+import com.microsoft.tfs.core.clients.versioncontrol.WorkspaceLocation;
 import com.microsoft.tfs.core.clients.versioncontrol.specs.version.ChangesetVersionSpec;
 import com.microsoft.tfs.core.clients.versioncontrol.specs.version.LatestVersionSpec;

@@ -57,7 +58,7 @@ public class GoTfsVersionControlClient {
     }

     public GoTfsWorkspace createWorkspace(String workspace) {
-        return new GoTfsWorkspace(client.createWorkspace(null, workspace, null, null, null, null));
+        return new GoTfsWorkspace(client.createWorkspace(null, workspace, null, WorkspaceLocation.SERVER, null, null));
     }

     public Changeset[] queryHistory(String projectPath, ChangesetVersionSpec uptoRevision, int revsToLoad) {
jsmythsci commented 5 years ago

@arvindsv Thank you very much for this. I grabbed both files and will test the build out over the next couple of days.

jsmythsci commented 5 years ago

@arvindsv I ran some tests today and the test build you provided seems to be working as expected.

Test Summary

Here is a summary of my test results:

  1. Build 18.11, bundled Java: bug exists.
  2. Build 18.12.0-8187 (test build), standalone Java: bug does not exist.
  3. Build 18.11, standalone Java: bug exists.

Test Details

My tests were performed on a new VM running Windows 2012 R2 using a snapshot to restore state between each test. Here is the process I followed:

  1. Confirm existing bug:
    1. Download and install go-server 18.11.
    2. Download and install go-agent 18.11.
    3. Connect agent to server.
    4. Create test pipeline as described in initial post.
    5. Run pipeline.
    6. Confirm pipeline fails during file checkout with error message documented in initial post.
  2. Test fix:
    1. Download and install Java 1.8.
    2. Download and run go-server (test version).
    3. Download and run go-agent (test version).
    4. Connect agent to server.
    5. Create test pipeline as described in initial post.
    6. Run pipeline.
    7. Confirm pipeline succeeds.
  3. Reconfirm bug using standalone Java:
    1. Download and install Java 1.8.
    2. Download and install go-server 18.11.
    3. Download and install go-agent 18.11.
    4. Connect agent to server.
    5. Create test pipeline as described in initial post.
    6. Run pipeline.
    7. Confirm pipeline fails during file checkout with error message documented in initial post.
jsmythsci commented 5 years ago

Regarding the impact of using server vs local workspaces, I believe the only technical difference between the two is whether or not the client downloads repository metadata to enable offline work.

My understanding of the limitations of TFS workspaces is that:

  1. A workspace owner/name combination must be unique: a single TFS user cannot have more than 1 TFS workspace with the same name.
  2. Workspace physical location (hostname + local path) must be unique across all of TFS: 2 users on a shared workstation, for example, cannot both work out of the same shared local directory, even if it maps to the same TFS location for both of them.

To the best of my knowledge workspace location (server vs local) is not a factor in any limitation so any potential for conflict between multiple agents running concurrently will likely exist regardless of workspace location.

arvindsv commented 5 years ago

@jsmythsci Thank you for checking that.

You might be right. Your knowledge of server workspace location is probably more up to date anyway. :) I'll read about it some more and think about the code a bit more. If I don't see anything that can go wrong, I might make the change. In the worst case, I'll probably make this an option that can be turned on. Either way, you should get this change in the 19.1 release in January. I won't be able to analyze it in time for the current release (this week).

arvindsv commented 5 years ago

Umm. Also, in case you don't see me say anything by first week of January or so, please feel free to remind me about this.

arvindsv commented 5 years ago

Looking at some articles such as this and this, it looks like server workspaces were the default in TFS 2010. TFS 2012 introduced local workspaces and it seems to be, as you said, all about being able to work offline.

My suspicion is that we were using server workspaces before the upgrade to TFS SDK 14 (since there was no such thing as a local workspace before). If that's the case, I'm less worried about making this change to use server workspaces.

Some things you might have to check, @jsmythsci, when you find the time:

  1. Ability to create a directory in the working directory.

  2. Ability to change an existing locally checked out file (might not be allowed, since it's a server workspace and it's supposed to be read-only).

  3. Making sure that a change in a file is seen by GoCD properly and the number of open workspaces as seen on the server (is there even such an ability?) doesn't keep increasing.

The last one: My thought is - a local workspace is like a full checkout. A server workspace is partial and is still "connected" to the server, in a sense. Does it cause extra load (mostly memory or filesystem) on the server?

jsmythsci commented 5 years ago

Thank you for your continued attention to this, @arvindsv.

I can confirm that GoCD uses server workspaces prior to the TFS SDK update. There was a bug (#2840) whereby agents were silently failing to delete their workspaces and when I check our TFS server for workspaces owned by our GoCD user I can see that they are all server workspaces. This visibility helped me when investigating for this current issue because, if I checked quickly enough, I could see that the workspaces created on newer versions of GoCD were local rather than server workspaces.

Here are my thoughts on the points you raise:

  1. I have not explicitly tested this but I believe it would work as expected. We have some workflows that depend on copying files into working directories after fetching materials and I can't see why creating a directory would be any different from creating a file in this context. If you think it's valuable then I would be happy to create a pipeline to explicitly test creating directories.
  2. When using server workspaces (GoCD v16.11) all files are checked out read-only and cannot be modified without removing the read-only attribute (Windows agent). I can see how this could be an issue for anyone who is using the current version of GoCD with TFS; hard-coding the use of a server workspace would cause errors if they have Tasks in their Jobs that modify files without first removing the read-only attribute. With that in mind it may be better to add a configuration parameter and retain the current default behaviour.
  3. I'm not sure what scenarios you are thinking of here. My understanding is that the GoCD server is responsible for detecting changes to files and that it polls the TFS server directly, without using a workspace. For agents, a workspace is created for each TFS material when a Stage is configured to fetch materials and is destroyed as soon as the files have been downloaded from TFS.
  4. (Unnumbered in your post.) I think that, given GoCD's workflow (create workspace -> get files -> delete workspace), extra load would be placed on the network and maybe the TFS server with a local workspace since the client (GoCD agent) must download repository metadata that is not required with a server workspace. I believe that any additional load caused by a server workspace would stem from the client's need to communicate with the server for every TFS operation but my understanding is that there are no such operations in GoCD's workflow.

Thank you again for your time and effort. Let me know your thoughts and if there are any tests you would like me to run.

arvindsv commented 5 years ago

@jsmythsci

  1. Yes, please. A confirmation, either way, should be fine. I would expect it works, as well.

  2. Ok.

  3. That's right. This change is supposed to only happen on the agent side. So, it should be fine. What I was thinking about was probably similar to #2840. We could check this scenario: Agent creates a server workspace. New commit happens. Agent uses the same server workspace and doesn't create another. I'm not sure how the connection to the TFS server works. If it's just a matter of having some metadata around to say that this is a server workspace, that's fine. If it's more, then it might be worth it to check that this change doesn't keep anything open on the TFS server side. I understand that the behavior will revert to whatever was happening in earlier versions of GoCD - and so, should be the same. I'm just taking this chance to document some of that, if possible.

  4. You're probably right about that.

I've opened https://github.com/gocd/gocd/pull/5608 which should allow me to make a simple change to put in a temporary toggle for this, controlled from the server side (assuming that change is accepted). Once the toggle is in place and works for a while, we can look for a more permanent solution (maybe by changing the TFS material config), unless we decide to default to server workspaces.

If you can check point (1) above, just for completeness' sake, that would be good. If possible, please check point (3) too. I understand it's more involved and if you cannot, that's ok. I'm asking only because you have seem to have easier access to a TFS server and probably know more about it than I do. :)

ketan commented 5 years ago

I feel that an agent should delete the server workspace at the end of a build. Not deleting would cause several server workspaces to be created (I believe there was a bug that we fixed regarding this.)

Also, this problem will just get worse with elastic agents, where a server workspaces will be created for every job that runs.

On Sat, Dec 29, 2018, 4:28 AM Aravind SV <notifications@github.com wrote:

@jsmythsci https://github.com/jsmythsci

1.

Yes, please. A confirmation, either way, should be fine. I would expect it works, as well. 2.

Ok. 3.

That's right. This change is supposed to only happen on the agent side. So, it should be fine. What I was thinking about was probably similar to #2840 https://github.com/gocd/gocd/issues/2840. We could check this scenario: Agent creates a server workspace. New commit happens. Agent uses the same server workspace and doesn't create another. I'm not sure how the connection to the TFS server works. If it's just a matter of having some metadata around to say that this is a server workspace, that's fine. If it's more, then it might be worth it to check that this change doesn't keep anything open on the TFS server side. I understand that the behavior will revert to whatever was happening in earlier versions of GoCD - and so, should be the same. I'm just taking this chance to document some of that, if possible. 4.

You're probably right about that.

I've opened #5608 https://github.com/gocd/gocd/pull/5608 which should allow me to make a simple change to put in a temporary toggle for this, controlled from the server side (assuming that change is accepted). Once the toggle is in place and works for a while, we can look for a more permanent solution (maybe by changing the TFS material config), unless we decide to default to server workspaces.

If you can check point (1) above, just for completeness' sake, that would be good. If possible, please check point (3) too. I understand it's more involved and if you cannot, that's ok. I'm asking only because you have seem to have easier access to a TFS server and probably know more about it than I do. :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/gocd/gocd/issues/5427#issuecomment-450440123, or mute the thread https://github.com/notifications/unsubscribe-auth/AAApZhVrDyHCiyeYFS-_EIxPVP5IWD06ks5u9qIugaJpZM4YsCQa .

arvindsv commented 5 years ago

Yes, we will need to understand what is happening first (and how a server workspace is "connected" to the TFS server - and what it leaves behind if it is not removed). Of course, removing it every time might also mean slower checkouts every time, etc.

jsmythsci commented 5 years ago

@arvindsv

For point (1) I created a simple pipeline to test and confirmed that creating a directory works as expected.

For point (3) I'm not sure the scenario you describe is valid under normal operating conditions. As seen in the log snippet included in alexbestul's excellent post on #2840, GoCD unmaps TFS workspaces as soon as it is done getting the files from the server. In the example alexbestul provided, the expected lifetime of the TFS workspace was only about 0.4 seconds.

I think it is important to note that, in TFS parlance, a "workspace" is not a collection of files and folders on a user's machine but rather a mapping between a collection of files and folders on a client machine and that same collection on the version control server.

I also think the term "location" is misleading, given what a workspace is. I can think of the following questions relating to the location of a TFS workspace:

I agree with @ketan that TFS workspaces (whether local or server) should not persist between job runs; this is how GoCD works today and I see no reason to change that.

arvindsv commented 5 years ago

@jsmythsci I forgot to mention that the latest experimental installers (19.1.0-8395) on https://www.gocd.org/download/ should have the changes for this. You'll need to set a system property as mentioned here: https://github.com/gocd/gocd/pull/5693

It'll also be released as a part of 19.1 and if all goes well, we can turn it on by default in 19.2 or 19.3 and remove the need for the system property soon after that.

rajiesh commented 5 years ago

Keeping this issue open for @jsmythsci to confirm if the fix helped solve the problem

jsmythsci commented 5 years ago

Sorry to take so long to follow up on this. Our work on GoCD stalled while waiting for version 19.01 and other projects took priority.

I just tested upgrading from 16.11 to 19.01 and my test builds are working as intended. This was unexpected, however, as I did not set the following system property, which I believed would be required:

gocd.agent.extra.properties=toggle.agent.tfs.use.server.workspace.location=Y

Was this done intentionally?

Either way, I'm glad that it is working for us now. I am eager to get our system current so we can start leveraging the improvements the GoCD team has made over the past 2 years.

arvindsv commented 5 years ago

@jsmythsci That's indeed strange. I double-checked the logic just now and it should be defaulting to using null for the server workspace location.

jsmythsci commented 5 years ago

@arvindsv Thank you for confirming.

I must have set the system property ahead of time on our test system.

I confirmed that removing it resulted in the same failures I initially reported and that when I added it back to the server configuration the builds completed as expected.

Thank you again for the help and support working through this issue with us.

arvindsv commented 5 years ago

@jsmythsci You're welcome. :) I'll setup a reminder to change the false to true by default, in a couple of releases and then remove the toggle altogether a couple of releases after that.

unkinected commented 5 years ago

Hey everyone. I just upgraded server and agents to 19.2 today and I'm still having this problem; when pointing to a single file for the material, the pipeline downloads a file named after the destination directory and breaks.

I see comments about needing to toggle a system problem, but I'm not quite sure what that means. I don't see anything like the above settings in the wrapper-server.conf file (I'm on Windows). Do I need to set an environment variable? What am I missing?

Thanks!

jsmythsci commented 5 years ago

Hello @unkinected,

To get this working there are 2 things that you need to do:

  1. The material's Destination Directory attribute must include a file name. It doesn't necessarily have to be the same as the file name in TFS but unless there is a good reason that it must be changed, it probably should be kept the same. This is a workaround for #183.
  2. The GoCD server needs to be started with the following system property set: gocd.agent.extra.properties=toggle.agent.tfs.use.server.workspace.location=Y

As documented here, the way to do this on Windows is to add a line like this to the wrapper-server.conf file:

wrapper.java.additional.XX=-Dgocd.agent.extra.properties=toggle.agent.tfs.use.server.workspace.location=Y

where XX represents the next wrapper.java.additional index.

Hope this helps.

unkinected commented 5 years ago

Thanks. I tried all that... if I do NOT change the destination directory to a file name, I get the same problem before but at least it doesn't error out. When I try to change the dest directory to a filename with no folder path, it works. However, if I specify a folder path, I get this message:

Invalid directory name '.\folder\filename.ext' It should be a valid relative path.

Mostly there, but this will do for me for now. Thanks!

jsmythsci commented 5 years ago

I think it might be a Windows vs Unix thing.

Try setting it to ./folder/filename.ext (forward slashes instead of backslashes) to see if that helps.

arvindsv commented 5 years ago

FYI: I've opened #6283 to change the default value of this to use SERVER.