Errors when migrating content to staging.cnx.org

[brittweinstein]

When trying to copy content from one server to another, we get this error: ERROR - Failed uploading module Introduction, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3595/m60207/sword

[reedstrm]

@brittweinstein Can we get a C&P of the terminal session? I haven't used that tool in a while, and am not sure how to reproduce.

[brittweinstein]

@reedstrm I'll email you the files here is the command I ran: ./content-copy.py -s Phase2test.json -i Phase2test.tsv -c -a 1

[mmulich]

@brittweinstein could you share that file with @reedstrm and I (pumazi) on share.cnx.org? I should probably take a stab at this before pulling @reedstrm away from his pre-planned work. Thanks.

[mmulich]

@brittweinstein can you point me to the content-copy-tool's source location? I found rhatpos/content-copy-tool, but I don't think that is the most up-to-date version.

[brittweinstein]

@pumazi that is the version that most of the CMs work on--the older one. You should be able to download the most updated files from here: https://github.com/Rhaptos/cnx-tools/tree/master/content-copy-tool We've had better luck with the non-updated tool, though. I uploaded the files to https://share.cnx.org/index.php/apps/files/?dir=%2FDocuments

[mmulich]

So far I’ve got ValidationError: The user (words_elemalg) does not exist. on dev. This could just be a difference between dev and staging.

[mmulich]

@brittweinstein Could it be that words_elemalg user doesn't exist on prod (and therefore staging)?

[brittweinstein]

@pumazi that's definitely likely! let me ask bblodget to try with accounts that exist on prod

[mmulich]

After I added the user to the site the tool completed without error. We could potentially add that user to staging if you'd like?

[Bkblodget]

When I initially ran the tool my json file never had anything pointing to the user words_elemalg. I had it referencing the username for this book that is currently used on prod: cnxelemalg. Attached is the json file used for migration and the tsv file. I just ran the migration again and received a problematic error, but I am unsure if it is an issue with the server or my migration tool.

I used the following command: ./content-copy.py -s Phase2test.json -i Phase2test.tsv -c -r

phase2migration.zip

[mmulich]

@Bkblodget, that's the same info I ran on dev. I do see the same errors in the server log too. The words_elemalg user isn't being used to make the transfer, but it does appear to be the owner of the content in the source site (legacy-staging1). Therefore, when applying the content to the destination site the attributed user must exist.

The logs do show that a few things went through though, so I'm a bit confused.

@Bkblodget get could you attach the relevant bits of your logs or the entire file. That would be the last parts of the content-copy.log file.

[openstaxalina]

@pumazi we migrate frequently from content staging servers, and the usernames on the content staging servers (for example, words_elemalg) are not required to exist on production for the migration to work. We designate a different (OpenStax) user who owns the modules on production. We need to be able to migrate to production without having the usernames from the content staging server on production.

[Bkblodget]

@pumazi I sent my files to @lwtchu and she was able to successfully migrate the majority of the content. It appears that the issue is mostly with my content tool specifically.

[lwtchu]

Here's the stuffs from my migration log. @pumazi "2017-02-23 15:10:22,150 - content-copy - DEBUG - Logger is up and running." "2017-02-23 15:10:22,180 - content-copy - DEBUG - Bookmap configuration has been created" "2017-02-23 15:10:22,181 - content-copy - DEBUG - Columns from input file: ['Unit Number', 'Chapter Number', 'Chapter Title', 'Module title', 'Development Module ID', 'Staging Module ID', 'Staging Workgroup']" "2017-02-23 15:10:22,188 - content-copy - DEBUG - Bookmap has been created" "2017-02-23 15:10:22,188 - content-copy - DEBUG - Copier has been created" "2017-02-23 15:10:22,188 - content-copy - DEBUG - Role configuration has been created." "2017-02-23 15:10:22,188 - content-copy - DEBUG - ContentCreator has been created." "2017-02-23 15:10:22,188 - content-copy - INFO - -------- Summary ---------------------------------------" "2017-02-23 15:10:22,188 - content-copy - INFO - Source: http://legacy-staging1.cnx.org" "2017-02-23 15:10:22,189 - content-copy - INFO - Destination: http://legacy-staging.cnx.org" "2017-02-23 15:10:22,189 - content-copy - INFO - User: cnxelemalg" "2017-02-23 15:10:22,189 - content-copy - INFO - Content: Phase2test" "2017-02-23 15:10:22,189 - content-copy - INFO - Which Chapters: 1, 2, 3, 4, 5" "2017-02-23 15:10:22,189 - content-copy - INFO - Number of Modules: 41" "2017-02-23 15:10:22,189 - content-copy - INFO - Create placeholders?: False" "2017-02-23 15:10:22,189 - content-copy - INFO - Copy content? True" "2017-02-23 15:10:22,189 - content-copy - INFO - Edit roles? True" "2017-02-23 15:10:22,189 - content-copy - INFO - Authors: cnxelemalg" "2017-02-23 15:10:22,189 - content-copy - INFO - Maintainers: cnxelemalg" "2017-02-23 15:10:22,190 - content-copy - INFO - Rightsholders: cnxelemalg" "2017-02-23 15:10:22,190 - content-copy - INFO - Create collections? False" "2017-02-23 15:10:22,190 - content-copy - INFO - Publish content? False" "2017-02-23 15:10:22,993 - content-copy - DEBUG - Beginning processing." "2017-02-23 15:10:22,993 - content-copy - INFO - Copying content for module: m10182 - 1.0 Introduction" "2017-02-23 15:10:30,397 - content-copy - INFO - Copying content for module: m10183 - 1.1 Introduction to Whole Numbers " "2017-02-23 15:11:01,593 - content-copy - INFO - Copying content for module: m10184 - 1.2 Use the Language of Algebra " "2017-02-23 15:11:44,687 - content-copy - INFO - Copying content for module: m10185 - 1.3 Add and Subtract Integers " "2017-02-23 15:12:34,733 - content-copy - INFO - Copying content for module: m10186 - 1.4 Multiply and Divide Integers" "2017-02-23 15:12:54,883 - content-copy - INFO - Copying content for module: m10187 - 1.5 Visualize Fractions" "2017-02-23 15:13:11,287 - content-copy - INFO - Copying content for module: m10188 - 1.6 Add and Subtract Fractions " "2017-02-23 15:13:43,192 - content-copy - INFO - Copying content for module: m10189 - 1.7 Decimals " "2017-02-23 15:14:28,535 - content-copy - INFO - Copying content for module: m10190 - 1.8 The Real Numbers" "2017-02-23 15:15:05,603 - content-copy - INFO - Copying content for module: m10191 - 1.9 Properties of Real Numbers" "2017-02-23 15:15:28,614 - content-copy - INFO - Copying content for module: m10192 - 1.10 Systems of Measurement" "2017-02-23 15:15:51,399 - content-copy - INFO - Copying content for module: m10214 - 2.0 Introduction" "2017-02-23 15:16:12,289 - content-copy - INFO - Copying content for module: m10215 - 2.1 Solve Equations Using the Subtraction and Addition Properties of Equality" "2017-02-23 15:16:56,979 - content-copy - INFO - Copying content for module: m10216 - 2.2 Solve Equations Using the Division and Multiplication Properties of Equality" "2017-02-23 15:21:24,559 - content-copy - ERROR - Failed uploading module Solve Equations Using the Division and Multiplication Properties of Equality, response 500 Internal Server Error when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3596/m60249/sword" "2017-02-23 15:21:24,559 - content-copy - INFO - Copying content for module: m10217 - 2.3 Solve Equations with Variables and Constants on Both Sides" "2017-02-23 15:26:17,857 - content-copy - ERROR - Failed uploading module Solve Equations with Variables and Constants on Both Sides, response 500 Internal Server Error when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3596/m60262/sword" "2017-02-23 15:26:17,859 - content-copy - INFO - Copying content for module: m10218 - 2.4 Use a General Strategy to Solve Linear Equations" "2017-02-23 15:28:58,476 - content-copy - ERROR - Failed uploading module Use a General Strategy to Solve Linear Equations, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3596/m60275/sword" "2017-02-23 15:28:58,477 - content-copy - INFO - Copying content for module: m10219 - 2.5 Solve Equations with Fractions or Decimals " "2017-02-23 15:30:33,931 - content-copy - INFO - Copying content for module: m10220 - 2.6 Solve a Formula for a Specific Variable" "2017-02-23 15:31:23,621 - content-copy - INFO - Copying content for module: m10221 - 2.7 Solve Linear Inequalities" "2017-02-23 15:35:05,639 - content-copy - ERROR - Failed uploading module Solve Linear Inequalities, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3596/m60270/sword" "2017-02-23 15:35:05,640 - content-copy - INFO - Copying content for module: m10193 - 3.0 Introduction" "2017-02-23 15:35:12,962 - content-copy - INFO - Copying content for module: m10194 - 3.1 Use a Problem-Solving Strategy " "2017-02-23 15:39:18,319 - content-copy - INFO - Copying content for module: m10195 - 3.2 Solve Percent Applications " "2017-02-23 15:39:56,919 - content-copy - ERROR - Failed uploading module Solve Percent Applications , response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3597/m60246/sword" "2017-02-23 15:39:56,920 - content-copy - INFO - Copying content for module: m10196 - 3.3 Solve Mixture Applications " "2017-02-23 15:41:03,057 - content-copy - INFO - Copying content for module: m10197 - 3.4 Solve Geometry Applications: Triangles, Rectangles and the Pythagorean Theorem" "2017-02-23 15:42:27,292 - content-copy - ERROR - Failed uploading module Solve Geometry Applications: Triangles, Rectangles and the Pythagorean Theorem, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3597/m60233/sword" "2017-02-23 15:42:27,292 - content-copy - INFO - Copying content for module: m10198 - 3.5 Solve Uniform Motion Applications" "2017-02-23 15:43:13,265 - content-copy - INFO - Copying content for module: m10199 - 3.6 Solve Applications with Linear Inequalities" "2017-02-23 15:43:24,376 - content-copy - INFO - Copying content for module: m10247 - 4.0 Introduction" "2017-02-23 15:43:29,533 - content-copy - INFO - Copying content for module: m10248 - 4.1 Use the Rectangular Coordinate System " "2017-02-23 15:44:47,255 - content-copy - INFO - Copying content for module: m10249 - 4.2 Graph Linear Equations in Two Variables " "2017-02-23 15:47:02,878 - content-copy - ERROR - Failed uploading module Graph Linear Equations in Two Variables , response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3598/m60273/sword" "2017-02-23 15:47:02,879 - content-copy - INFO - Copying content for module: m10250 - 4.3 Graph with Intercepts" "2017-02-23 15:47:58,235 - content-copy - INFO - Copying content for module: m10251 - 4.4 Understand Slope of a Line " "2017-02-23 15:49:47,927 - content-copy - INFO - Copying content for module: m10252 - 4.5 Use the Slope-Intercept Form of an Equation of a Line" "2017-02-23 15:51:35,777 - content-copy - INFO - Copying content for module: m10253 - 4.6 Find the Equation of a Line" "2017-02-23 15:53:41,629 - content-copy - ERROR - Failed uploading module Find the Equation of a Line, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3598/m60267/sword" "2017-02-23 15:53:41,630 - content-copy - INFO - Copying content for module: m10254 - 4.7 Graphs of Linear Inequalities" "2017-02-23 15:56:51,773 - content-copy - ERROR - Failed uploading module Graphs of Linear Inequalities, response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3598/m60315/sword" "2017-02-23 15:56:51,774 - content-copy - INFO - Copying content for module: m10200 - 5.0 Introduction" "2017-02-23 15:56:57,177 - content-copy - INFO - Copying content for module: m10201 - 5.1 Solving Systems of Equations by Graphing" "2017-02-23 15:58:32,663 - content-copy - INFO - Copying content for module: m10202 - 5.2 Solving Systems of Equations by Substitution" "2017-02-23 15:59:51,144 - content-copy - INFO - Copying content for module: m10203 - 5.3 Solving Systems of Equations by Elimination" "2017-02-23 16:00:33,854 - content-copy - INFO - Copying content for module: m10204 - 5.4 Solve Applications with Systems of Equations" "2017-02-23 16:01:43,588 - content-copy - INFO - Copying content for module: m10205 - 5.5 Solve Mixture Applications with Systems of Equations" "2017-02-23 16:03:33,338 - content-copy - INFO - Copying content for module: m10206 - 5.6 Graphing Systems of Linear Inequalities" "2017-02-23 16:05:23,966 - content-copy - DEBUG - Finished copying content." "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 2.2 Solve Equations Using the Division and Multiplication Properties of Equality" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 2.3 Solve Equations with Variables and Constants on Both Sides" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 2.4 Use a General Strategy to Solve Linear Equations" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 2.7 Solve Linear Inequalities" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 3.2 Solve Percent Applications " "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 3.4 Solve Geometry Applications: Triangles, Rectangles and the Pythagorean Theorem" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 4.2 Graph Linear Equations in Two Variables " "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 4.6 Find the Equation of a Line" "2017-02-23 16:05:23,967 - content-copy - ERROR - Failed uploading module - 4.7 Graphs of Linear Inequalities" "2017-02-23 16:05:23,967 - content-copy - INFO - ------- Process completed --------"

[kerwinso]

@pumazi Dogpile! I was also asked by Bey to try the content copy operation on my machine with her command and files mentioned in this comment.

I got failures on ~35% of the modules, which is significant. All of the failed modules returned 400 Bad Request errors, but I did not get 'Problematic Errors' on any of them. Here's the relevant lines from my log file: KMoney-content-copy022317.txt

The only thing I can think of is if there is a file upload size limit on legacy-staging (the destination server). However, the 14 or so modules that failed for me ran the gamut in file size, from <2MB to 97MB.

Looks like Larissa had about 9 modules fail due to 400 errors, although they weren't the same modules that failed for me. I just think it's strange that we're all getting such different results. We're all running the same version of the tool (it hasn't been updated in 2 years), which makes me think it may be something with the server configuration for legacy-staging.

[dtwilliamson]

Default client_max_body_size for file uploads to nginx is 1M, if that's pertinent.

[mmulich]

The first time I ran the tool I pointed it at a specific instance. My last attempt to run the tool was against the load balancer. Only 7/10 items were successful.

I'm seeing the following traceback with some degree of repetition as we continue to do these content-copy-tool tries.

Traceback (most recent call last):
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/Shared/DC/ZRDB/Connection.py", line 71, in __setstate__
    try: self.connect(self.connection_string)
  File "/var/lib/cnx/cnx-buildout/eggs/Products.ZPsycopgDA-2.4.6_rhaptos-py2.4.egg/Products/ZPsycopgDA/DA.py", line 127, in connect
    self._v_database_connection.open()
  File "/var/lib/cnx/cnx-buildout/eggs/Products.ZPsycopgDA-2.4.6_rhaptos-py2.4.egg/Products/ZPsycopgDA/db.py", line 96, in open
    self.getconn()
  File "/var/lib/cnx/cnx-buildout/eggs/Products.ZPsycopgDA-2.4.6_rhaptos-py2.4.egg/Products/ZPsycopgDA/db.py", line 58, in getconn
    conn.set_session(isolation_level=int(self.tilevel))
ProgrammingError: set_session cannot be used inside a transaction```

[mmulich]

@dtwilliamson @DarkPrinceFrost : Can you do a search on the production logs to see if ProgrammingError: set_session cannot be used inside a transaction shows up anywhere?

[DarkPrinceFrost]

Unable to find similar entries on foutain, quill, or gladius

[mmulich]

Here is another error that seems to be reoccurring:

2017-02-23T21:56:51 ERROR rhaptos.swordservice.plone.browser.sword Traceback (most recent call last):
  File "/var/lib/cnx/cnx-buildout/eggs/rhaptos.swordservice.plone-1.1-py2.4.egg/rhaptos/swordservice/plone/browser/sword.py", line 71, in wrapper
    value = func(*args, **kwargs)
  File "/var/lib/cnx/cnx-buildout/eggs/rhaptos.swordservice.plone-1.1-py2.4.egg/rhaptos/swordservice/plone/browser/sword.py", line 104, in __call__
    return self._handlePost()
  File "/var/lib/cnx/cnx-buildout/eggs/rhaptos.swordservice.plone-1.1-py2.4.egg/rhaptos/swordservice/plone/browser/sword.py", line 121, in _handlePost
    return adapter._handlePost()
  File "/var/lib/cnx/cnx-buildout/eggs/Products.RhaptosSword-2.3.2dev-py2.4.egg/Products/RhaptosSword/browser/views.py", line 122, in _handlePost
    cksum, merge == 'http://purl.org/oerpub/semantics/Merge')
  File "/var/lib/cnx/cnx-buildout/eggs/Products.RhaptosSword-2.3.2dev-py2.4.egg/Products/RhaptosSword/adapters.py", line 486, in updateContent
    makeContent(obj, subobjs)
  File "/var/lib/cnx/cnx-buildout/eggs/Products.CNXMLTransforms-0.23-py2.4.egg/Products/CNXMLTransforms/helpers.py", line 47, in makeContent
    obj.getPrimaryField().getMutator(obj)(body)
  File "/var/lib/cnx/cnx-buildout/parts/plone/ATContentTypes/content/image.py", line 131, in setImage
    self._setATCTFileContent(value, **kwargs)
  File "/var/lib/cnx/cnx-buildout/parts/plone/ATContentTypes/content/base.py", line 471, in _setATCTFileContent
    field.set(self, value, **kwargs) # set is ok
  File "/var/lib/cnx/cnx-buildout/parts/plone/Archetypes/Field.py", line 2100, in set
    instance=instance, **kwargs)
  File "/var/lib/cnx/cnx-buildout/parts/plone/Archetypes/Field.py", line 903, in _process_input
    file.manage_upload(value)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/OFS/Image.py", line 468, in manage_upload
    data, size = self._read_data(file)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/OFS/Image.py", line 517, in _read_data
    transaction.savepoint(optimistic=True)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/transaction/_manager.py", line 110, in savepoint
    return self.get().savepoint(optimistic)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/transaction/_transaction.py", line 295, in savepoint
    self._saveCommitishError() # reraises!
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/transaction/_transaction.py", line 292, in savepoint
    savepoint = Savepoint(self, optimistic, *self._resources)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/transaction/_transaction.py", line 675, in __init__
    savepoint = savepoint()
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/ZODB/Connection.py", line 1026, in savepoint
    self._commit(None)
  File "/var/lib/cnx/cnx-buildout/parts/zope2/lib/python/ZODB/Connection.py", line 530, in _commit
    raise ConflictError(object=obj)
ConflictError: database conflict error (oid 0x02d7cd3f, class Products.RhaptosModuleEditor.ModuleEditor.ModuleEditor)```

[mmulich]

When I run the content-copy-tool with the given data the console output of content-copy 2017-02-24 10:14:47 ERROR - Failed uploading module Decimals , response 400 Bad Request when sending to https://legacy-staging.cnx.org/GroupWorkspaces/wg3595/m60202/sword directly corresponds with the server log output of ValidationError: The user (words_elemalg) does not exist. (the full error previously pasted).

[mmulich]

Everything succeeded for me after having @dtwilliamson add the words_elemalg user to staging.

Can you guys please retest? Please ping me on slack before you do, so that I know when to be looking at the logs. (CC @brittweinstein @Bkblodget @kerwinso )

[kerwinso]

@pumazi Test succeeded with no errors.