Closed Derek-Hardy closed 3 years ago
curl -X POST localhost:8484/v1/projects/tm-obj-v6-test:import \
-H 'Content-Type: application/json' \
-d '{"input_url":"gs://v6_migration_db/2021-03-06T13:53:55_30312/2021-03-06T13:53:55_30312.overall_export_metadata"}'
gs://...
has no meaning when run locally. What you want to do is downloading that backup snapshot and changing the input_url
such that it points to your download location.
Using Testing Approach 2: Set up two separate versions of TEAMMATES on the staging server (one using Objectify 5 and the other using Objectify 6) for testing in both directions.
Backward:
Forward:
Approach 2: (backward - test on v5 data)
env: objectify-v6-migration
Error:
SEVERE: LoadException caught by WebApiServlet:
com.googlecode.objectify.LoadException: Error loading Key{projectId=tm-obj-v6-test, namespace=,
path=[PathElement{kind=FeedbackSession, id=null, name=non visible session%idOfTypicalCourse1}]}:
At path 'instructions': Expected value of type [STRING, LONG, DOUBLE, BOOLEAN],
got ENTITY: EntityValue{valueType=ENTITY, excludeFromIndexes=false, meaning=0,
value=FullEntity{key=null, properties={value=StringValue{valueType=STRING, excludeFromIndexes=true,
meaning=0, value=Please please fill in the following questions.}}}}
SEVERE: LoadException caught by WebApiServlet:
com.googlecode.objectify.LoadException: Error loading Key{projectId=tm-obj-v6-test, namespace=,
path=[PathElement{kind=FeedbackResponse, id=null,
name=ahBufnRtLW9iai12Ni10ZXN0ch0LEhBGZWVkYmFja1F1ZXN0aW9uGICAgNj4qOALDA%IFRes.fred.g@gmail.tmt%IFRes.emily.f@gmail.tmt}]}:
Expected type ENTITY at path 'answer' but instead found STRING
UPDATE:
FeedbackSession
, Instructor
on GCPanswer
to StringBoth errors seem to suggest that the data in the datastore might be in a format incompatible with the intended pojo field. I think it might be because currently we don't have any logic implemented to translate Text
from the old api to String
?
Environment: objectify-v6-migration
Ran into a similar error while running the dev server, which might give some clue on the problem.
The error is resolved locally, after deleting the local_db.bin
file which contains the data stored in the emulator.
Mar 14, 2021 5:29:03 PM teammates.common.util.Logger info
INFO: Request received: [GET] http://localhost:8080/webapi/courses, Params: {"entitytype":"instructor","coursestatus":"active"}, Headers: {"Origin":"http://localhost:4200","Cookie":"users=O%3A4%3A%22User%22%3A2%3A%7Bs%3A15%3A%22%00User%00userlevel%22%3Bi%3A10%3Bs%3A14%3A%22%00User%00username%22%3Bs%3A8%3A%22John+Doe%22%3B%7D%3Cx%3EO%3A4%3A%22User%22%3A2%3A%7Bs%3A15%3A%22%00User%00userlevel%22%3Bi%3A33%3Bs%3A14%3A%22%00User%00username%22%3Bs%3A12%3A%22Peter+Parker%22%3B%7D%3Cx%3EO%3A4%3A%22User%22%3A2%3A%7Bs%3A15%3A%22%00User%00userlevel%22%3Bi%3A87%3Bs%3A14%3A%22%00User%00username%22%3Bs%3A11%3A%22Gabe+Newell%22%3B%7D%80%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%00%08%F8%3Cx%3EO%3A4%3A%22User%22%3A2%3A%7Bs%3A15%3A%22%00User%00userlevel%22%3Bi%3A999%3Bs%3A14%3A%22%00User%00username%22%3Bs%3A7%3A%22jhacker%22%3B%7D; Idea-109df5e7=52a6823a-3d13-4d9e-ac0b-912744b25635; JSESSIONID=t9slz83CvFxe29J_Ot-X7w.node0; CSRF-TOKEN=E99D7CA9CD3A3A613F4D64188EFCBE862CD08476CFEF6878E44C7BE1AADFF4F9; dev_appserver_login=test@example.com:true:185804764220139124118","Accept":"application/json, text/plain, */*","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36","Referer":"http://localhost:4200/","Sec-Fetch-Site":"same-site","Sec-Fetch-Dest":"empty","Host":"localhost:8080","Sec-Fetch-Mode":"cors","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","ngsw-bypass":"true","Accept-Language":"en-US,en;q=0.9"}, Request ID: 604dd75f000631f0b1d57811
Mar 14, 2021 5:29:03 PM teammates.common.util.Logger severe
SEVERE: LoadException caught by WebApiServlet:
com.googlecode.objectify.LoadException: Error loading Key{projectId=teammates-john, namespace=, path=[PathElement{kind=Instructor, id=null, name=jh@gmail.comm%jh.gma-demo0}]}:
At path 'instructorPrivilegesAsText': Expected value of type [STRING, LONG, DOUBLE, BOOLEAN], got ENTITY: EntityValue{valueType=ENTITY, excludeFromIndexes=false, meaning=0, value=FullEntity{key=null, properties={value=StringValue{valueType=STRING, excludeFromIndexes=false, meaning=0, value={
"courseLevel": {
"canviewstudentinsection": true,
"cansubmitsessioninsection": true,
"canmodifysessioncommentinsection": true,
"canmodifycourse": true,
"canviewsessioninsection": true,
"canmodifysession": true,
"canmodifystudent": true,
"canmodifyinstructor": true
},
"sectionLevel": {},
"sessionLevel": {}
}}}}}
at com.googlecode.objectify.impl.EntityMetadata.load(EntityMetadata.java:84)
at com.googlecode.objectify.impl.LoadEngine.load(LoadEngine.java:196)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:154)
at com.googlecode.objectify.impl.LoadEngine$1.nowUncached(LoadEngine.java:140)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.impl.Round$1.nowUncached(Round.java:66)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.impl.HybridQueryResults.lambda$load$1(HybridQueryResults.java:88)
at com.google.common.collect.Iterators$6.transform(Iterators.java:783)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at com.google.common.collect.Iterators$ConcatenatedIterator.next(Iterators.java:1364)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:636)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at com.googlecode.objectify.impl.HybridQueryResults.hasNext(HybridQueryResults.java:93)
at com.google.common.collect.Iterators.addAll(Iterators.java:355)
at com.google.common.collect.Lists.newArrayList(Lists.java:143)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:22)
at com.googlecode.objectify.util.MakeListResult.translate(MakeListResult.java:12)
at com.googlecode.objectify.util.ResultTranslator.nowUncached(ResultTranslator.java:21)
at com.googlecode.objectify.util.ResultCache.now(ResultCache.java:30)
at com.googlecode.objectify.util.ResultProxy.invoke(ResultProxy.java:32)
at com.sun.proxy.$Proxy15.iterator(Unknown Source)
at teammates.storage.api.EntitiesDb.makeAttributes(EntitiesDb.java:199)
at teammates.storage.api.InstructorsDb.getInstructorsForGoogleId(InstructorsDb.java:164)
at teammates.logic.core.InstructorsLogic.getInstructorsForGoogleId(InstructorsLogic.java:131)
at teammates.logic.api.Logic.getInstructorsForGoogleId(Logic.java:215)
at teammates.ui.webapi.GetCoursesAction.getInstructorCourses(GetCoursesAction.java:66)
at teammates.ui.webapi.GetCoursesAction.execute(GetCoursesAction.java:46)
at teammates.ui.webapi.GetCoursesAction.execute(GetCoursesAction.java:22)
at teammates.ui.webapi.WebApiServlet.invokeServlet(WebApiServlet.java:93)
at teammates.ui.webapi.WebApiServlet.doGet(WebApiServlet.java:42)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at teammates.ui.webapi.OriginCheckFilter.doFilter(OriginCheckFilter.java:98)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:134)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.tools.development.jetty9.StaticFileFilter.doFilter(StaticFileFilter.java:123)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.google.appengine.tools.development.DevAppServerRequestLogFilter.doFilter(DevAppServerRequestLogFilter.java:28)
...
com.google.appengine.tools.development.jetty9.DevAppEngineWebAppContext.doScope(DevAppEngineWebAppContext.java:94)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.google.appengine.tools.development.jetty9.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:599)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: At path 'instructorPrivilegesAsText': Expected value of type [STRING, LONG, DOUBLE, BOOLEAN], got ENTITY: EntityValue{valueType=ENTITY, excludeFromIndexes=false, meaning=0, value=FullEntity{key=null, properties={value=StringValue{valueType=STRING, excludeFromIndexes=false, meaning=0, value={
"courseLevel": {
"canviewstudentinsection": true,
"cansubmitsessioninsection": true,
"canmodifysessioncommentinsection": true,
"canmodifycourse": true,
"canviewsessioninsection": true,
"canmodifysession": true,
"canmodifystudent": true,
"canmodifyinstructor": true
},
"sectionLevel": {},
"sessionLevel": {}
}}}}}
at com.googlecode.objectify.impl.Path.throwIllegalState(Path.java:102)
at com.googlecode.objectify.impl.translate.ValueTranslator.loadSafe(ValueTranslator.java:56)
at com.googlecode.objectify.impl.translate.NullSafeTranslator.load(NullSafeTranslator.java:21)
at com.googlecode.objectify.impl.PropertyPopulator.setValue(PropertyPopulator.java:95)
at com.googlecode.objectify.impl.PropertyPopulator.load(PropertyPopulator.java:55)
at com.googlecode.objectify.impl.translate.ClassPopulator.load(ClassPopulator.java:119)
at com.googlecode.objectify.impl.translate.ClassTranslator.loadSafe(ClassTranslator.java:113)
at com.googlecode.objectify.impl.translate.NullSafeTranslator.load(NullSafeTranslator.java:21)
at com.googlecode.objectify.impl.EntityMetadata.load(EntityMetadata.java:80)
... 99 more
Mar 14, 2021 5:29:23 PM com.google.appengine.api.datastore.dev.LocalDatastoreService$11 run
INFO: Time to persist datastore: 617 ms
On google cloud console, I can see that instructorPrivilegesAsText
(same for instructions
, answer
) are stored as Embedded entity
in json format instead of String
.
Conclusion:
Backward compatibility test
env: objectify-v6-migration
branch
Backend exceptions:
WARNING: UnauthorizedAccessException caught by WebApiServlet:
teammates.common.exception.UnauthorizedAccessException: Current account cannot access to courses of request entity type
at teammates.ui.webapi.GetCoursesAction.checkSpecificAccessControl(GetCoursesAction.java:35)
at teammates.ui.webapi.Action.checkAccessControl(Action.java:105)
WARNING: UnauthorizedAccessException caught by WebApiServlet:
teammates.common.exception.UnauthorizedAccessException: Course [tm.e2e.ICEdit.CS2104] is not accessible to instructor [ICEdit.observer@gmail.tmt] for privilege [canmodifyinstructor]
at teammates.logic.api.GateKeeper.verifyAccessible(GateKeeper.java:177)
at teammates.ui.webapi.CreateInstructorAction.checkSpecificAccessControl(CreateInstructorAction.java:38)
WARNING: UnauthorizedAccessException caught by WebApiServlet:
teammates.common.exception.UnauthorizedAccessException: You are not allowed to view this resource!
at teammates.ui.webapi.GetStudentAction.checkSpecificAccessControl(GetStudentAction.java:42)
WARNING: InvalidHttpRequestBodyException caught by WebApiServlet:
teammates.common.exception.InvalidHttpRequestBodyException: "invalidemail" is not acceptable to TEAMMATES as a/an email because it is not in the correct format. An email address contains some text followed by one '@' sign followed by some more text, and should end with a top level domain address like .com. It cannot be longer than 254 characters, cannot be empty and cannot contain spaces.
at teammates.ui.request.BasicRequest.assertTrue(BasicRequest.java:20)
at teammates.ui.request.AccountCreateRequest.validate(AccountCreateRequest.java:65)
at teammates.ui.webapi.Action.getAndValidateRequestBody(Action.java:216)
at teammates.ui.webapi.CreateAccountAction.execute(CreateAccountAction.java:30)
WARNING: InvalidHttpRequestBodyException caught by WebApiServlet:
teammates.common.exception.InvalidHttpRequestBodyException: [Please distribute all the points for distribution questions. To skip a distribution question, leave the boxes blank.]
at teammates.ui.webapi.SubmitFeedbackResponsesAction.execute(SubmitFeedbackResponsesAction.java:194)
Failed tests: (28)
org.gradle.internal.serialize.PlaceholderException: no such element: Unable to locate element: {"method":"css selector","selector":"#course\-id"}
org.openqa.selenium.NoSuchElementException at FeedbackSubmitPageE2ETest.java:57
org.gradle.internal.serialize.PlaceholderException: no such element: Unable to locate element: {"method":"css selector","selector":"#course\-name"}
org.openqa.selenium.NoSuchElementException at StudentCourseDetailsPageE2ETest.java:40
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at StudentHomePageE2ETest.java:37
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at FeedbackConstSumOptionQuestionE2ETest.java:47
java.lang.AssertionError: expected:<1> but was:<0>
at org.testng.AssertJUnit.fail(AssertJUnit.java:47)
at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:330)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:66)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:213)
at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:218)
at teammates.e2e.pageobjects.InstructorCoursesPage.verifyNumDeletedCourses(InstructorCoursesPage.java:121)
at teammates.e2e.cases.InstructorCoursesPageE2ETest.testAll(InstructorCoursesPageE2ETest.java:114)
java.lang.AssertionError: expected:<4> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at teammates.e2e.pageobjects.InstructorCourseDetailsPage.verifyNumStudents(InstructorCourseDetailsPage.java:64)
at teammates.e2e.cases.InstructorCourseDetailsPageE2ETest.testAll(InstructorCourseDetailsPageE2ETest.java:68)
java.lang.AssertionError: expected:<2> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at teammates.e2e.pageobjects.InstructorSearchPage.verifyStudentDetails(InstructorSearchPage.java:97)
at teammates.e2e.cases.InstructorSearchPageE2ETest.testAll(InstructorSearchPageE2ETest.java:74)
java.lang.NullPointerException at FeedbackMcqQuestionE2ETest.java:103
java.lang.NullPointerException at FeedbackRankRecipientQuestionE2ETest.java:102
java.lang.NullPointerException at AdminSearchPageE2ETest.java:46
java.lang.NullPointerException at FeedbackRubricQuestionE2ETest.java:115
...
Relevant code:
// GetStudentAction.java
private static final String UNAUTHORIZED_ACCESS = "You are not allowed to view this resource!";
@Override
void checkSpecificAccessControl() {
String courseId = getNonNullRequestParamValue(Const.ParamsNames.COURSE_ID);
CourseAttributes course = logic.getCourse(courseId);
StudentAttributes student;
String studentEmail = getRequestParamValue(Const.ParamsNames.STUDENT_EMAIL);
String regKey = getRequestParamValue(Const.ParamsNames.REGKEY);
if (studentEmail != null) {
student = logic.getStudentForEmail(courseId, studentEmail);
if (student == null || userInfo == null || !userInfo.isInstructor) {
throw new UnauthorizedAccessException(UNAUTHORIZED_ACCESS);
}
InstructorAttributes instructor = logic.getInstructorForGoogleId(courseId, userInfo.id);
gateKeeper.verifyAccessible(instructor, logic.getCourse(courseId), student.section,
Const.InstructorPermissions.CAN_VIEW_STUDENT_IN_SECTIONS);
} else if (regKey != null) {
getUnregisteredStudent().orElseThrow(() -> new UnauthorizedAccessException(UNAUTHORIZED_ACCESS));
} else {
if (userInfo == null || !userInfo.isStudent) {
throw new UnauthorizedAccessException(UNAUTHORIZED_ACCESS);
}
student = logic.getStudentForGoogleId(courseId, userInfo.id);
gateKeeper.verifyAccessible(student, course);
}
}
// GetCourseAction.java
@Override
void checkSpecificAccessControl() {
String entityType = getNonNullRequestParamValue(Const.ParamsNames.ENTITY_TYPE);
if (!((entityType.equals(Const.EntityType.STUDENT) && userInfo.isStudent)
|| (entityType.equals(Const.EntityType.INSTRUCTOR) && userInfo.isInstructor))) {
throw new UnauthorizedAccessException("Current account cannot access to courses of request entity type");
}
}
// GateKeeper.java
public void verifyAccessible(InstructorAttributes instructor, CourseAttributes course) {
verifyNotNull(instructor, "instructor");
verifyNotNull(instructor.courseId, "instructor's course ID");
verifyNotNull(course, "course");
verifyNotNull(course.getId(), "course ID");
if (!instructor.courseId.equals(course.getId())) {
throw new UnauthorizedAccessException("Course [" + course.getId() + "] is not accessible to instructor ["
+ instructor.email + "]");
}
}
Work is done
Environment:
release
branchobjectify-v6-migration
branchTesting approach 1: connect dev server to the staging server datastore
build.properties
the attributeapp.production.gcs.bucketname=tm-obj-v6-test.appspot.com
contextInitialized
inOfyHelper
as below:setupLocalDatastoreHelper
andtearDownLocalDatastoreHelper
inBaseE2ETestCase
classTesting approach 2: export data from staging data buckets and import into emulator
gsutil -m cp -r "gs://v6_migration_db/2021-03-06T13:53:55_30312/" <dst_url>
gcloud beta emulators datastore start --host-port=localhost:8484 --consistency=1.0
⚠️ NOTE: actual local folder/file name might not display the same as in
input_url
path.