search

jfrog / nexus2artifactory

NexusToArtifactory - A tool designed to ease migration from Sonatype Nexus to JFrog Artifactory.
Apache License 2.0
67 stars 54 forks source link

sec group #21

Closed jenkinsworld closed 6 years ago

jenkinsworld commented 6 years ago

Hello, We have installed nexus2artifactory tool on server where Nexus is installed. Python version is 2.6. But we are getting error "Configuration file security.xml is not valid."

Nexus path provided is: sonatype-work\nexus (where exactly conf folder is available and inside it security.xml is available)

Could you please help us with above error.

Regards, Chandrasekhar

DarthFennec commented 6 years ago

Hi, thanks for opening an issue. If you run the tool with the options -v debug -l logfile.txt, it will create a debug log file. That should provide more information about what exactly it's seeing as invalid.

jenkinsworld commented 6 years ago

Hello Travis Foster, Thanks for prompt reply. Yes i have tried with options -v debug -l logfile.txt. Please find the below error from logfile.txt file. Could you please check and suggest how to fix the error.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 2018-04-24 05:26:03,667 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'ENTER' pressed. 2018-04-24 05:26:03,681 [MainThread] [INFO] (nex2art.core.Nexus2:28) - Reading Nexus config from /data/nexus/sonatype-work/nexus/conf/nexus.xml. 2018-04-24 05:26:03,705 [MainThread] [INFO] (nex2art.core.Nexus2:58) - Successfully read Nexus config. 2018-04-24 05:26:03,705 [MainThread] [INFO] (nex2art.core.Ldap2:17) - Reading LDAP config from /data/nexus/sonatype-work/nexus/conf/ldap.xml. 2018-04-24 05:26:03,709 [MainThread] [INFO] (nex2art.core.Ldap2:21) - Successfully read LDAP config. 2018-04-24 05:26:03,710 [MainThread] [INFO] (nex2art.core.Security2:22) - Reading security config from /data/nexus/sonatype-work/nexus/conf/security.xml. 2018-04-24 05:26:03,710 [MainThread] [ERROR] (nex2art.core.Security2:27) - inside try 2018-04-24 05:26:19,759 [MainThread] [ERROR] (nex2art.core.Security2:55) - Error reading security config: Traceback (most recent call last): File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nex2art/core/Security2.py", line 38, in refresh self.flattentargets(privs) File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nex2art/core/Security2.py", line 89, in flattentargets targ = priv['target'] KeyError: 'target' 2018-04-24 05:26:19,760 [MainThread] [INFO] (nex2art.core.Format:124) - Updating data tree with Nexus data. 2018-04-24 05:26:19,762 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/Initial Setup/Nexus Data Directory'. 2018-04-24 05:26:27,568 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'h' pressed. 2018-04-24 05:26:36,143 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'q' pressed. 2018-04-24 05:26:39,471 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'q' pressed. 2018-04-24 05:26:39,474 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/'. 2018-04-24 05:26:39,476 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/Initial Setup'. 2018-04-24 05:26:50,591 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'q' pressed. 2018-04-24 05:26:56,208 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'y' pressed. 2018-04-24 05:26:56,210 [MainThread] [INFO] (root:33) - Terminating Nexus migration tool. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Regards, Chandrasekhar

DarthFennec commented 6 years ago

It seems you have a privilege which uses a repository target that doesn't exist. Can you confirm?

This appears to be the same as one of the errors from issue #19.

jenkinsworld commented 6 years ago

Hello Travis Foster,

We have multiple repository targets available (In fact they had added long back)in our Nexus instance. Could you please help me to identify which repository target has causing this issue.

Regards, Chandrasekhar

DarthFennec commented 6 years ago

Assuming you have too many privileges to effectively look through them one at a time, I've pushed an update to the tool that adds some new debug logs. If you get the latest version and run the tool again, it should log each privilege name as it discovers them, so the one right before the error should be the permission that's causing the issue.

jenkinsworld commented 6 years ago

Hello Travis Foster,

Thanks for quick response. Currently facing some git related issues while updating the tool. Will share you the logs asap.

Regards, Chandrasekhar

jenkinsworld commented 6 years ago

Hello,

I have updated the tool and re-run the tool for more logs as you suggested. please find the error below ############################################################### 2018-05-09 08:31:01,199 [MainThread] [DEBUG] (nex2art.core.Security2:213) - Extracting privilege snapshots - (update) 2018-05-09 08:31:01,199 [MainThread] [DEBUG] (nex2art.core.Security2:213) - Extracting privilege snapshots - (delete) 2018-05-09 08:31:01,199 [MainThread] [DEBUG] (nex2art.core.Security2:86) - Flattening repository target into privilege All npm Repositories 2018-05-09 08:31:01,199 [MainThread] [DEBUG] (nex2art.core.Security2:86) - Flattening repository target into privilege All Rubygems Repositories 2018-05-09 08:31:01,200 [MainThread] [DEBUG] (nex2art.core.Security2:86) - Flattening repository target into privilege comamadeusccp 2018-05-09 08:31:01,200 [MainThread] [ERROR] (nex2art.core.Security2:51) - Error reading security config: Traceback (most recent call last): File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Security2.py", line 37, in refresh self.flattentargets(privs) File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Security2.py", line 87, in flattentargets targ = priv['target'] KeyError: 'target' 2018-05-09 08:31:01,201 [MainThread] [INFO] (nex2art.core.Format:124) - Updating data tree with Nexus data. 2018-05-09 08:31:01,202 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/Initial Setup/Nexus Data Directory'. 2018-05-09 08:31:08,285 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'q' pressed. 2018-05-09 08:31:08,287 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/'. 2018-05-09 08:31:08,290 [MainThread] [INFO] (nex2art.core.Validate:24) - Validating tree from '/Initial Setup'. 2018-05-09 08:31:09,245 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'q' pressed. 2018-05-09 08:31:13,004 [MainThread] [DEBUG] (nex2art.core.Screen:138) - Key 'y' pressed. 2018-05-09 08:31:13,006 [MainThread] [INFO] (root:33) - Terminating Nexus migration tool. ###########################################################################

Regards, Chandrasekhar

DarthFennec commented 6 years ago

So then it looks like the privilege in question is comamadeusccp. If you check which repository target that uses, you can check if that target exists.

jenkinsworld commented 6 years ago

Hi, We have repository target named comamadeusccp and the path it is pointing have artifacts present in it. And also it has all required privileges like create, read, delete and update.

Regards, Chandrasekhar

DarthFennec commented 6 years ago

So to clarify:

Is that correct?

In that case, something really weird must be going on ... I think I would check the XML files to make sure everything is in order.

You'll find your privileges in security.xml. It looks like this:

  <privileges>
    <!-- ... -->
    <privilege>
      <id>4e3a68ab46e</id>
      <name>comamadeusccp - (create)</name>
      <description>a description here</description>
      <type>target</type>
      <properties>
        <property>
          <key>repositoryTargetId</key>
          <value>255d710c5ff</value>
        </property>
        <property>
          <key>method</key>
          <value>create,read</value>
        </property>
        <!-- ... -->
      </properties>
    </privilege>
    <!-- ... -->
  </privileges>

The important things to note here are the <name> field (make sure you're looking at the right privileges), and the <properties>. Specifically, the <property> with a <key> of repositoryTargetId. The <value> here is the id of the repository target, which is usually a random hex string (in my case it's 255d710c5ff, but yours is most certainly different.

First, ensure that all of your comamadeusccp - (...) privileges have the same repositoryTargetId.

Second, we want to ensure that this is the actual id specified by the repository target. You'll find repository targets in the nexus.xml file. It looks like this:

  <repositoryTargets>
    <!-- ... -->
    <repositoryTarget>
      <id>255d710c5ff</id>
      <name>comamadeusccp</name>
      <contentClass>maven2</contentClass>
      <patterns>
        <!-- ... -->
      </patterns>
    </repositoryTarget>
    <!-- ... -->
  </repositoryTargets>

Just make sure that the <id> field matches the repositoryTargetId property from your permissions.

If everything matches up right, then I'm really not sure what's going on. Try providing your XML files (or if that's not possible, at least a minimum verifiable example), so that I can try to reproduce the issue myself. Thanks.

jenkinsworld commented 6 years ago

Hello,

The reposiotoryGroupid is empty for comamadeusccp repositoryTarget. please refer below for same. And also there is no comamadeusccp repositoryTarget in nexus.xml as well.

!!!!!Please note i have mentioned id as some random number below. image

Is it something problem with creation of previleges for repositoryTarget : comamadeusccp ??

Regards, Chandrasekhar

DarthFennec commented 6 years ago

It's fine that there are no values for repositoryGroupId or repositoryId, that just means the privilege can apply to any repository rather than one in particular. There does seem to be a value for repositoryTargetId, and that value is the one you want to look for in your nexus.xml. It might not have the name comamadeusccp.

If there is no <repositoryTarget> entry in nexus.xml with the same <id> found in this repositoryTargetId field, that is the source of the issue. I would recommend finding the repository target this privilege is supposed to be configured with (or creating it if it doesn't exist), and inserting its id here. Alternatively, if this privilege isn't particularly important, you can always delete it from Nexus, which will prevent the issue from occurring. I would generally expect this to not be an option. However, if the target doesn't exist or is not linked properly, that would mean the privilege is not applying anyway, and if it was in use I would expect this problem to have been discovered before now.

jenkinsworld commented 6 years ago

Hello,

I could able to remove errors regarding repository target now clearing all unused/non existing repo targets in this Nexus instance.

Now I am getting below error, could you please check let me know what is wrong. ############################################################################# 2018-08-16 14:11:54,835 [MainThread] [ERROR] (nex2art.core.Security2:77) - Unable to convert regexes for repository target 3, Lookarounds are not yet supported: Traceback (most recent call last): File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Security2.py", line 71, in gettargets pospats, negpats = parser.convert(target['patterns']) File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Pattern.py", line 11, in convert for regex in regexes: inter.extend(parser.parseRegex(regex).convert()) File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Pattern.py", line 540, in convert for opt in chunk.convert(): File "/data/home/bdsadmin/Artifactory/nexus2artifactory/nexus2artifactory/nex2art/core/Pattern.py", line 534, in convert raise RuntimeError("Lookarounds are not yet supported") RuntimeError: Lookarounds are not yet supported #########################################################

DarthFennec commented 6 years ago

To describe which files within a repository a privilege applies to, Nexus uses a regular expression syntax, whereas Artifactory uses a more straightforward wildcard pattern syntax. The migrator will attempt to convert from one to the other automatically, whenever possible. However, the conversion code is relatively new, and is incomplete: there are a number of more advanced regex features that it doesn't yet support. This includes lookarounds (positive/negative lookaheads/lookbehinds). You're getting this error because one of your repository targets specifies a regex containing a lookaround.

The error is just warning you that the regex couldn't be automatically converted. This shouldn't cause the program to crash or anything, it just means that if you want to migrate that particular permission, you'll need to set the wildcard patterns manually through the migrator's UI.

jenkinsworld commented 6 years ago

Could you please help me to get it done and solve this error.? Is there a way to convert them automatically from any other tool/method??

DarthFennec commented 6 years ago

As I said, you'll need to convert them manually through the UI. After opening the tool and connecting to Nexus, go into the security menu, and then go into the permissions menu, and look for permissions with a red exclamation point. It will tell you what the original regex is, and all you have to do is decide on an appropriate set of wildcard patterns with the same meaning. Enter those into the include and exclude pattern fields. Then you should be able to migrate.