Closed mmullerbr closed 7 years ago
@amp5208
@tabladrum: Question regarding the Dockerfile configurations - can I assume that your team has tested environment variable additions through a Docker container to the Feature Collectors? I ask because my code does not explicitly read system environment variable declarations, unless Java Springboot takes care of that for me.
@mmullerbr , Thank you for reaching out. Pending @tabladrum's answer above, please confirm the following parts of your configuration:
JIRA_BASE_URL=[your jira url, assuming that your Java keystore is updated with your SSL cert for jira]
JIRA_CREDENTIALS=[jira credentials with the following access format, in base64: user:password]
# Also, you are missing your Jira field mappings to one of the following internal statuses: todo, doing, done. Examples:
feature.todoStatuses[0]=Open
feature.todoStatuses[1]=Groom
feature.todoStatuses[2]=Selected for Development
feature.todoStatuses[3]=Backlog
feature.todoStatuses[4]=Grooming
feature.todoStatuses[5]=ToDo
feature.todoStatuses[6]=To Do
feature.todoStatuses[6]=Backlog1
feature.doingStatuses[0]=Validation
feature.doingStatuses[1]=Test
feature.doingStatuses[2]=In Process
feature.doingStatuses[3]=In Progress
feature.doingStatuses[4]=Awaiting Approval - 2
feature.doingStatuses[5]=Change Request - 2
feature.doneStatuses[0]=Awaiting Approval
feature.doneStatuses[1]=Done
feature.doneStatuses[2]=Resolved
Please check the above configurations and let me know if you are still having issues.
@jimzucker - Can you see this if you have time? Thanks
@amp5208 @tabladrum Yes the Env variables set in the docker file & docker-compose files pass thru correctly. The spring classes are taking care of pulling them in and making them available internally. If you have a specific issue let us know.
@jimzucker , Thank you for confirming.
In that case, @mmullerbr , please see my updated and simplified application properties configuration details at the following README fork; try providing ALL of the properties that are relevant and not commented out via Docker ENVs: https://github.com/amp5208/Hygieia/blob/master/jira-feature-collector/README.md#sample-applicationproperties-file-with-minimum-overrides. I also provided explanations for most properties.
Just saw this as I was going to issues page. I was able to get the most recent jira collector working today (built off of master). I ran into a similar issue and found that I was missing some properties. Try comparing the properties that are generated in the docker container to this (my ordering is a little different):
#Database Name
database=dashboard
#Database HostName - default is localhost
dbhost=192.168.33.11
#Database Port - default is 27017
dbport=27017
#Database Username - default is blank
dbusername=db
dbpassword=dbpass
#Collector schedule (required)
feature.cron=0 0/5 * * * *
#Page size for data calls (Jira maxes at 1000)
feature.pageSize=1000
#In-built folder housing prepared REST queries (required)
feature.queryFolder=jiraapi-queries
#Jira API Query file names (String template requires the files to have .st extension) (required)
feature.storyQuery=story
feature.epicQuery=epic
feature.projectQuery=projectinfo
feature.memberQuery=memberinfo
feature.sprintQuery=sprintinfo
feature.teamQuery=teaminfo
feature.trendingQuery=trendinginfo
#Jira Connection Details
feature.jiraProxyUrl=
feature.jiraProxyPort=
# Trending Query: Number of days in a sprint (not-required)
feature.sprintDays=60
# Trending Query: Length of sprint week (not-required)
feature.sprintEndPrior=7
#Scheduled Job prior minutes to recover data created during execution time (usually, 2 minutes is enough)
feature.scheduledPriorMin=2
#Delta change date that modulates the collector item task - should be about as far back as possible, in ISO format (required)
feature.deltaCollectorItemStartDate=2008-01-01T00:00:00.000000
#Jira Connection Details
feature.jiraBaseUrl=<not shown>
feature.jiraQueryEndpoint=rest/api/2/
#reference: https://www.base64decode.org/
#feature.jiraOauthAuthtoken=sdfghjkl==
#feature.jiraOauthRefreshtoken=sdfagheh==
#feature.jiraOauthRedirecturi=uri.this.is.test:uri
#feature.jiraOauthExpiretime=234567890987
#Start dates from which to begin collector data, if no other data is present - usually, a month back is appropriate (required)
feature.deltaStartDate=2015-03-01T00:00:00.000000
feature.masterStartDate=2008-01-01T00:00:00.000000
# In Jira, general IssueType IDs are associated to various "issue"
# attributes. However, there is one attribute which this collector's
# queries rely on that change between different instantiations of Jira.
# Please provide a numerical ID reference to your instance's IssueType for
# the lowest level of Issues (e.g., "user story") specific to your Jira
# instance. Note: You can retrieve your instance's IssueType ID
# listings via the following URI: https://[your-jira-domain-name]/rest/api/2/issuetype/
feature.jiraIssueTypeId=6
# In Jira, your instance will have its own custom field created for "sprint" or "timebox" details,
# which includes a list of information. This field allows you to specify that data field for your
# instance of Jira. Note: You can retrieve your instance's sprint data field name
# via the following URI, and look for a package name com.atlassian.greenhopper.service.sprint.Sprint;
# your custom field name describes the values in this field:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
feature.jiraSprintDataFieldName=customfield_10005
# In Jira, your instance will have its own custom field created for "super story" or "epic" back-end ID,
# which includes a list of information. This field allows you to specify that data field for your instance
# of Jira. Note: You can retrieve your instance's epic ID field name via the following URI where your
# queried user story issue has a super issue (e.g., epic) tied to it; your custom field name describes the
# epic value you expect to see, and is the only field that does this for a given issue:
# https://[your-jira-domain-name]/rest/api/2/issue/[some-issue-name]
feature.jiraEpicIdFieldName=customfield_10006
#Core Artifacts
# These arrays of status mappings must be completed in order for custom source system statuses to be providable by
# Hygieia. Currently, Hygieia only maps to the following 3 status mappings: to do, doing, and done. You can find
# all available issue status mappings by going to your instance's version of the following API call:
# http://fake.jira.com/rest/api/2/status/. Your statuses are listed in the JSON response as "name."
feature.todoStatuses[0]=OPEN
feature.doingStatuses[0]=IN PROGRESS
feature.doingStatuses[1]=TEST
feature.doingStatuses[2]=TESTING
feature.doneStatuses[0]=DONE
feature.doneStatuses[1]=CLOSED
feature.jiraCredentials=<not shown>
logging.level.com.capitalone.dashboard.collector=DEBUG
logging.level.com.atlassian.jira=INFO
#logging.level.org.apache.http.impl.nio.client.DefaultHttpAsyncClient=DEBUG
logging.level.com.capitalone.dashboard=DEBUG
#logging.level.org.springframework=DEBUG
@MarkRx , Thank you for following up regarding this issue. I also see some issues with your property configuration. See the following for changes:
feature.jiraIssueTypeId=6 # --> Should be a string value that corresponds to the given ID, e.g., "Story" or "Epic"
# Leave the following blanked-out
feature.jiraOauthAuthtoken=
feature.jiraOauthRefreshtoken=
feature.jiraOauthRedirecturi=
feature.jiraOauthExpiretime=
# Comment these lines out if you have no proxy:
#feature.jiraProxyUrl=
#feature.jiraProxyPort=
# Reduce your page size:
feature.pageSize=100
Essentially, make your property configs/environment settings for Docker match that of the property configs on this README: https://github.com/amp5208/Hygieia/blob/master/jira-feature-collector/README.md#sample-applicationproperties-file-with-minimum-overrides
@amp5208 I had originally commented out the proxy configuration but If I remember correctly that resulted in a NPE at FeatureDataClientSetupImpl:87:
if (!this.featureSettings.getJiraProxyUrl().isEmpty()
Hi, Guys! Thaks for the feedback!
@amp5208, I have tried adding the fields mapping before, but this time I realized that after a docker restart (in order to restart the collector and get the properties) all changes made manually in the properties file are wiped out. (Is there a way to add more properties via docker configuration or should I open another issue for this?)
In order to work around this, I had to stop the collector process (killing it forces a Docker stop/start and consequently the additional properties are wiped out) and run a new collector process (maybe I'm missing something, because I'm not a Linux expert).
By doing this the collector runs without errors.
nohup java -jar jira-feature-collector.jar --spring.config.location=hygieia-jira-feature-collector-new.properties &
2016-06-02 16:21:03,039 INFO c.c.d.collector.FeatureCollectorTask - Feature Data Collection Finished 2016-06-02 16:22:00,000 INFO c.c.d.collector.CollectorTask - Running Collector: Jira 2016-06-02 16:22:00,003 INFO c.c.d.collector.FeatureCollectorTask - Starting Feature collection... 2016-06-02 16:22:00,015 INFO c.c.d.c.team.TeamDataClientSetupImpl - Beginning collection of team data at Thu Jun 02 16:22:00 UTC 2016 2016-06-02 16:22:01,006 INFO c.c.d.c.team.TeamDataClientSetupImpl - Process took :0.990285674 seconds to update 2016-06-02 16:22:01,012 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Beginning collection of project data at Thu Jun 02 16:22:01 UTC 2016 2016-06-02 16:22:01,979 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Process took :0.966078813 seconds to update 2016-06-02 16:22:01,998 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Beginning collection of feature data at Thu Jun 02 16:22:01 UTC 2016 2016-06-02 16:22:02,984 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Process took :0.986238778 seconds to update 2016-06-02 16:22:02,984 INFO c.c.d.collector.FeatureCollectorTask - Feature Data Collection Finished
However, no data is being presented on the dashboard. Here follows my new properties file:
database=dashboard dbhost=172.17.0.2 dbport=27017 dbusername=db dbpassword=dbpass
feature.pageSize=100
feature.deltaStartDate=2016-03-01T00:00:00.000000 feature.masterStartDate=2016-03-01T00:00:00.000000 feature.deltaCollectorItemStartDate=2016-03-01T00:00:00.000000
feature.cron=0 * * * * *
feature.queryFolder=jiraapi-queries feature.storyQuery=story feature.epicQuery=epic
feature.jiraProxyUrl= feature.jiraProxyPort=
feature.jiraBaseUrl=https://myJira.atlassian.net feature.jiraQueryEndpoint=rest/api/2/ feature.jiraCredentials=asasasasasasasasasas
feature.jiraOauthAuthtoken= feature.jiraOauthRefreshtoken= feature.jiraOauthRedirecturi= feature.jiraOauthExpiretime=
feature.jiraIssueTypeId=Story feature.jiraSprintDataFieldName=customfield_10017 feature.jiraEpicIdFieldName=customfield_10018
feature.todoStatuses[0]=To Do feature.doingStatuses[0]=In Progress feature.doneStatuses[0]=Done
Thanks in advance,
@MarkRx : Did you receive this NullPointerException with the newest version on upstream master? I ask because issues related to the proxy configuration were fixed in a previous issue/PR (https://github.com/capitalone/Hygieia/pull/322)
@mmullerbr : There is, in fact, a different way to pass in the property file for Hygieia. You may use the Docker -v [dockerhost dir]:[dockercontainer dir]
to volume map to a local property file configuration from within the Docker container; more details can be found on the Docker official content page: Volume Mapping. Additionally, you may have to force a different CMD
run option for the Docker container, so that it knows where to find your application.properties file; this can be done by adding a new java command at run time at the end of the docker run
command: java -jar jira-feature-collector.jar --spring.config.location=hygieia-jira-feature-collector-new.properties
. Please see below for a sample command / setup:
#!/bin/bash
docker run -d -p 8080:8080 -v $SRC_DIR/hygieia-jira-feature-collector-new.properties:$DOCKER_CNTR_DIR/hygieia-jira-feature-collector-new.properties $HYGIEIA_DOCKER_IMAGE_TAG:latest kill $(pgrep -o java) && java -jar jira-feature-collector.jar --spring.config.location=$DOCKER_CNTR_DIR/hygieia-jira-feature-collector-new.properties
@jimzucker : Please verify my above statements and make corrections as needed.
I just submitted PR #614 will make using external custom configs easier.
@amp5208 Yes. Here is the stack trace:
2016-06-03 15:06:00,067 ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
at com.capitalone.dashboard.client.team.TeamDataClientSetupImpl.updateObjectInformation(TeamDataClientSetupImpl.java:85) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
at com.capitalone.dashboard.client.team.TeamDataClientImpl.updateTeamInformation(TeamDataClientImpl.java:130) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:99) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
at com.capitalone.dashboard.collector.FeatureCollectorTask.collect(FeatureCollectorTask.java:26) ~[jira-feature-collector.jar!/:2.0.2-SNAPSHOT]
at com.capitalone.dashboard.collector.CollectorTask.run(CollectorTask.java:63) ~[core-2.0.2-SNAPSHOT.jar!/:2.0.2-SNAPSHOT]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_74]
at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) [na:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_74]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_74]
@mmullerbrhttps://github.com/mmullerbr - The standard docker pattern is to create a docker-compose.override.yml and change/define whatever you want there. This will change what is defined in the config file after you run docker compose.
Here is the context of my file that I used for testing:
hygieia-github-scm-collector: environment:
hygieia-sonar-codequality-collector: environment:
@MarkRx , Thank you for following up with the stack trace, and thank you @jimzucker for the Docker compose expertise.
It seems that you are receiving the error likely because the proxy setting variables are, in fact null, not blank. the line of code affected here is this: https://github.com/capitalone/Hygieia/blob/master/jira-feature-collector/src/main/java/com/capitalone/dashboard/client/team/TeamDataClientSetupImpl.java#L85. I will check this on my next commit and submit a PR if needed. For now, setting proxy to blank. i.e.:
feature.jiraProxyUrl=
feature.jiraProxyPort=
... should do the trick. Let me know if you are still seeing odd behavior.
@jimzucker, the jira-properties-builder.sh is not prepared to receive field mapping form docker-compose file and replace them in the properties file. I was able to modify the jira-properties-builder.sh inside the container and get the collector up and running with the properties file presented below. However I will loose my changes every time a container recreation is needed - this is a problem for another day... :)
Unfortunately my dash board remains not presenting sprints and tasks information. Any other thoughts?
database=dashboard dbhost=172.17.0.2 dbport=27017 dbusername=db dbpassword=dbpass
feature.cron=0 * * * * * feature.pageSize=100
feature.queryFolder=jiraapi-queries feature.storyQuery=story feature.epicQuery=epic
feature.jiraProxyUrl= feature.jiraProxyPort=
feature.jiraBaseUrl=https://myJira.atlassian.net/ feature.jiraQueryEndpoint=rest/api/2/ feature.jiraCredentials=asasasasasasasa
feature.jiraOauthAuthtoken= feature.jiraOauthRefreshtoken= feature.jiraOauthRedirecturi= feature.jiraOauthExpiretime=
feature.deltaCollectorItemStartDate=2016-01-01T00:00:00.000000 feature.deltaStartDate=2016-01-01T00:00:00.000000 feature.masterStartDate=2016-01-01T00:00:00.000000
feature.jiraIssueTypeId=Story feature.jiraSprintDataFieldName=customfield_10017 feature.jiraEpicIdFieldName=customfield_10018
feature.todoStatuses[0]=To Do feature.doingStatuses[0]=In Progress feature.doneStatuses[0]=Done
@mmullerbr I am not clear on the problem can you give me a simple example?
@mmullerbr , At this point, I will need some stack traces and/or UI console logs for further debug, as your property configurations appear correct at face-value. Please provide those at your convenience.
Closed. Reopen if there are any problems. Use latest codebase and follow cleanup instructions provided in https://github.com/capitalone/Hygieia/pull/1137.
Hi, guys!
I'm facing an error when running the collector: ERROR c.c.d.c.s.FeatureDataClientSetupImpl - Unexpected error in Jira paging request of java.lang.NullPointerException.
The project name is displayed in the Jira dashboard configuration dropbox, but no other information is presented after I select one project.
This is my collector log: 2016-06-01 13:44:00,016 INFO c.c.d.collector.FeatureCollectorTask - Starting Feature collection... 2016-06-01 13:44:00,054 INFO c.c.d.c.team.TeamDataClientSetupImpl - Beginning collection of team data at Wed Jun 01 13:44:00 UTC 2016 2016-06-01 13:44:01,143 INFO c.c.d.c.team.TeamDataClientSetupImpl - Process took :1.08712382 seconds to update 2016-06-01 13:44:01,150 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Beginning collection of project data at Wed Jun 01 13:44:01 UTC 2016 2016-06-01 13:44:02,040 INFO c.c.d.c.p.ProjectDataClientSetupImpl - Process took :0.887490468 seconds to update 2016-06-01 13:44:02,045 INFO c.c.d.c.s.FeatureDataClientSetupImpl - Beginning collection of feature data at Wed Jun 01 13:44:02 UTC 2016 2016-06-01 13:44:02,960 ERROR c.c.d.c.s.FeatureDataClientSetupImpl - Unexpected error in Jira paging request of java.lang.NullPointerException [null]
This is my docker-compose Jira configuration: hygieia-jira-feature-collector: image: hygieia-jira-feature-collector:latest container_name: hygieia-jira volumes:
./logs:/hygieia/logs
links:
hygieia-api
environment:
JIRA_ISSUE_TYPE_ID=Story #also tried 10001
{
}
JIRA_SPRINT_DATA_FIELD_NAME=customfield_10017
customfield_10017:["com.atlassian.greenhopper.service.sprint.Sprint@f33f15 ..."]
JIRA_EPIC_FIELD_NAME=customfield_10018
customfield_10018: "JDOS-1" #My Epic Name
Thanks in advance,